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Abstract 

This  thesis  examines  the  impact  that  recommendations  made  by  the 
President’s  Blue  Ribbon  Commission  on  Defense  Management,  informally 
known  as  the  Packard  Commission,  had  on  reducing  cost  overruns  in  major  DoD 
acquisition  programs.  Cost  overruns  are  a  recurring  problem  in  the  DoD,  and  the 
study  of  possible  effects  resulting  from  the  implementation  of  acquisition  reform 
efforts  such  as  the  Packard  Commission  study  could  alter  this  trend.  In  this  era 
of  acquisition  reform  and  downsizing  it’s  important  that  policy  makers  understand 
the  effects  past  and  current  policies  have  had  and  are  having  on  reducing  the 
ever  present  problem  of  cost  overruns.  Conclusions  drawn  in  this  thesis  may 
guide  and  direct  DoD  policy  makers  in  drafting  future  regulations  and  policies. 

This  study  examined  269  contracts  completed  between  January  1,  1988 
and  December  31,1 995.  It  was  found  that  cost  performance  for  contracts 
completed  after  the  recommendations  went  into  effect  was  poorer  than  cost 
performance  prior  to  the  change.  It  was  also  found  that  a  more  significant 
difference  occurred  between  contracts  in  development  phases  than  those  in 
production  phases.  In  fact,  percentage  cost  overruns  for  development  contracts 
nearly  tripled  after  the  policy  went  into  effect.  Possible  explanations  and 
implications  of  this  discovery  are  provided. 
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THE  IMPACT  OF  THE  PACKARD  COMMISSION’S 


RECOMMENDATIONS  ON  REDUCING  COST  OVERRUNS 
IN  MAJOR  DEFENSE  ACQUISITION  PROGRAMS 


I.  Introduction 


General  Issue 

The  phenomenon  of  cost  overruns  in  Department  of  Defense  acquisition 
programs  has  been  a  problem  for  decades.  In  fact,  regulations  to  control 
defense  procurements  extend  as  far  back  as  the  1 940’s.  However,  these 
policies  have  accomplished  little  in  controlling  or  reducing  the  problem.  A  1993 
Rand  Corporation  study  discussed  the  extent  of  cost  growth  in  the  DoD  and 
indicated  that  cost  growth  has  fluctuated  around  20  percent  since  the  mid  1960’s 
and  that  little  improvement  has  occurred  overtime  (Dreznerand  others,  1993:2). 
Other  research  indicates  that  the  average  cost  overrun  on  DoD  acquisition 
contracts  is  approximately  40  percent  (Gansler,  1989:4).  However  one 
measures  the  unplanned  cost  increases  (growth  or  overruns  to  be  explained 
later  in  this  thesis)  is  academic;  the  magnitude  of  the  problem  persists  and  is 
readily  seen  as  20  to  40  percent  can  represent  a  notable  loss.  In  the  1980’s, 
when  public  pressure  on  controlling  defense  spending  was  at  its  peak,  critics 
estimated  that  in  one  year  alone  sloppiness,  duplication,  and  fraud  could  add  up 
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to  $23  billion  which  was  about  one  third  of  total  weapon  procurement 
expenditures  at  that  time  (Griffiths,  1985:144).  While  procurement  expenditures 
in  the  1990’s  are  less  than  half  of  the  1980’s  levels,  the  magnitude  of  DoD 
procurement  activities  remains  significant,  and  over  the  last  two  decades  it 
seems  one  is  constantly  hearing  through  the  media  that  some  defense  program 
will  cost  taxpayers  millions  of  dollars  more  than  was  budgeted. 

President  Reagan  established  the  Packard  Commission  in  1986  in  an 
effort  to  combat  the  still  pervasive  inefficiencies  in  the  DoD’s  procurement 
system  and  to  stimulate  procurement  reform.  The  group  was  officially  called  the 
President’s  Blue  Ribbon  Commission  on  Defense  Management  but  was 
informally  named  the  Packard  Commission  after  its  chairman,  industrialist  and 
former  Deputy  Secretary  of  Defense  David  Packard.  They  examined  DoD 
management  in  general  but  concentrated  particularly  on  the  acquisition  process. 
The  Commission  concluded  that  the  primary  problems  with  the  acquisition 
process  were  the  same  inefficiencies  of  the  60’s  and  70’s,  namely  cost  growth, 
schedule  delays,  and  performance  shortfalls.  Furthermore,  they  believed  these 
inefficiencies  were  actually  the  problems  and  not  just  symptoms.  In  response 
the  Commission  issued  a  number  of  recommendations  to  alleviate  these 
problems. 
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Background 

Unplanned  cost  increases  in  the  DoD’s  weapon  system  procurements  can 
escalate  to  staggering  amounts  and  can  impact  not  only  the  operational  users 
but  the  people  of  this  country  as  well.  Furthermore,  cost  growth  can  affect  our 
national  security.  “For  a  long  period  of  time,  the  government  has  been  trying  to 
reduce  cost  growth  and  increase  visibility  over  defense  acquisition  programs” 
(AFSCP  173-4,  1989:5).  In  fact,  over  the  last  25  years  the  DoD  has 
implemented  over  20  significant  regulatory  and  administrative  initiatives  that 
were  intended  to  improve  cost  performance  in  weapon  system  development 
(Drezner  and  others,  1993:28-29).  Unfortunately,  cost  growth,  perceived  as  an 
inefficiency  in  the  procurement  process,  continued  throughout  the  1970’s  and 
1980’s.  What  was  thought  to  have  been  controlled  in  the  1970’s  (cost  growth) 
resurfaced  as  a  pressing  problem  in  the  1980’s.  In  fact,  cost  growth  in  the 
1980’s  seemed  to  be  increasing  (Singer,  1983:2;  Drezner  and  others,  1993:30). 

It  appears  that  none  of  the  regulatory  and  administrative  initiatives  implemented 
in  the  60’s,  70’s,  or  early  80’s  were  successful  in  reducing  cost  growth  in  major 
defense  acquisition  contracts. 

Statement  of  the  Problem 

The  presence  of  cost  overruns  is  a  recurring  problem  in  the  DoD,  and  the 
study  of  possible  effects  resulting  from  the  implementation  of  acquisition  reform 
efforts  such  as  the  Packard  Commission  study  could  alter  this  trend.  In  this  era 
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of  acquisition  reform  and  downsizing  it  is  important  that  policy  makers 
understand  the  effects  past  and  current  policies  have  had  and  are  having  on  the 
ever  present  problem  of  cost  overruns.  The  overall  goal  of  this  research  effort  is 
to  provide  the  DoD  with  an  evaluation  of  a  top  level  acquisition  policy.  This 
evaluation  could  then  serve  as  a  guide  for  policy  makers  by  offering  a  critical 
evaluation  of  the  current  path  of  acquisition  policy — specifically  whether  or  not 
the  DoD  is  heading  in  the  “right”  direction  with  regards  to  it’s  acquisition  policies. 
Furthermore,  this  evaluation  may  reveal  whether  or  not  DoD  acquisition  policies 
are  targeting  the  actual  problems. 

Answers  to  issues  such  as  these  would  yield  tremendous  insight  as  to  the 
appropriateness  of  current  policies  and  the  potential  effectiveness  of  future 
policies.  Conclusions  drawn  in  this  thesis  may  guide  and  direct  DoD  policy 
makers  in  the  drafting  of  future  regulations  and  policies. 

The  specific  objective  of  this  research  is  to  test  the  effectiveness  of  the 
Packard  Commission’s  recommendations  on  reducing  cost  overruns  in 
Department  of  Defense  acquisition  programs.  The  research  questions  to  be 
answered  and  the  specific  hypothesis  to  be  tested  are  as  follows: 

1 .  Is  the  mean  final  overrun  percentage  (FO%)  for  contracts  completed 
before  Dec.31,  1991  different  than  the  mean  percentage  afterwards?  This  will 
be  determined  by  testing  the  following  null  hypothesis: 
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H1 :  Given  a  sample  of  completed  contracts,  the  final  overrun  percentage 
for  contracts  completed  after  Dec.  31 ,  1991  is  the  same  as  the  percent 
overrun  of  contracts  completed  before  that  date. 

2.  Does  the  mean  FO%  for  contracts  completed  before  and  after  Dec.31. 
1991  differ  between  program  phase?  This  will  be  determined  by  testing  the 
following  null  hypotheses: 

H2:  Given  a  sample  of  completed  development  contracts,  the  final  overrun 
percentage  for  contracts  completed  after  Dec.  31,  1991  is  the  same  as  the 
percent  overrun  of  contracts  completed  before  that  date. 

H3:  Given  a  sample  of  completed  production  contracts,  the  final  overrun 
percentage  for  contracts  completed  after  Dec.  31,  is  the  same  as  the  percent 
overrun  of  contracts  completed  before  that  date. 

Currently  there  appear  to  be  two  schools  of  thought  on  the  proper 
measurement  of  unplanned  cost  increases:  cost  growth  and  cost  overruns. 
Several  studies  have  used  the  terms  interchangeably,  but  they  are  in  fact 
referring  to  different  measurements.  While  recognizing  the  contributions  made 
by  both  measurements,  this  thesis  distinguishes  between  the  two  and  builds 
upon  the  cost  overrun  model  as  the  measuring  tool  throughout  this  study.  Also, 
this  thesis  measures  percentage  overruns  rather  than  gross  overruns  as  the 
unadjusted  measure  fails  to  consider  contract  size  (in  terms  of  dollar  value)  and 
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inflation.  By  converting  the  unadjusted  final  overrun  figure  to  a  final  overrun 
percentage  one  can  adjust  the  figure  for  size  and  inflation  thus  providing  a 
relative  measure  of  cost  performance. 

Scope  of  the  Study 

The  principal  statistical  analysis  used  in  this  study  tests  for  differences  in 
the  population  means  of  percentage  overruns  of  contracts  completed  before  and 
after  the  policy  changes  implemented  from  the  Commission’s  recommendations. 
The  procedure  used  to  measure  and  compare  the  distributions  of  data  will  be 
discussed  in  Chapter  3.  Using  the  sample  mean  as  an  estimator  of  the 
population  mean  allows  for  inferences  to  be  made  about  the  differences  between 
population  parameters. 

The  method  applied  in  this  study  relies  on  cost  performance  data 
contained  in  the  Defense  Acquisition  Executive  Summary  (DAES)  database. 

This  database  contains  cost  performance  data  on  completed  contracts  extending 
back  as  far  as  1977  from  every  branch  of  the  service.  The  database  used  in  this 
study  contains  cost  information  from  378  programs  encompassing  1 ,843 
individual  contracts.  Because  of  its  broad  and  extensive  coverage  the  database 
can  provide  program  and  researchers  with  a  valuable  management  tool 
highlighting  “both  potential  and  actual  program  problems  to  the  Under  Secretary 
of  Defense  (Acquisition  and  Technology)  before  they  become  significant”  (DODI 
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5000.2,  1996; Part  6).  The  DAES  database  available  for  this  study  contains  cost 
performance  data  from  1977  through  1995. 

Organization  of  the  Study 

This  thesis  is  organized  in  the  traditional  format.  The  following  chapter 
introduces  supportive  information  and  related  studies  in  order  to  build  a 
foundation  upon  which  the  remainder  of  the  thesis  rests.  That  chapter  will  also 
answer  the  following  investigative  questions: 

•  What  is  the  significance  of  cost  overruns  to  the  Department  of 
Defense? 

•  What  factors  are  believed  to  cause  cost  overruns  in  the  Department  of 
Defense? 

•  How  are  cost  overruns  measured? 

•  What  has  been  the  DoD’s  response  to  cost  overruns  over  the  years? 

Subsequent  chapters  will  then  discuss  the  methodology  used  to  test  the  three 
hypothesis,  will  present  the  results  of  that  analysis,  and  will  discuss  the  findings 
of  this  research  effort  as  they  apply  to  the  Department  of  Defense.  Finally, 
recommendations  will  be  made  for  further  research. 
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II.  Literature  Review 


Chapter  Overview 

Researchers  have  conducted  numerous  studies  over  the  last  30  years  in 
an  effort  to  capture  the  driving  force  behind  a  cost  growth  “phenomenon”.  These 
studies  have  attempted  to  quantify,  predict,  measure,  and  control  cost  growth, 
yet  current  research  leads  us  to  believe  that  our  predictions  have  failed  as  have 
our  most  of  our  attempts  to  control  it.  While  some  studies  may  have  produced 
more  effective  models  than  others  the  bottom  line  remains... we  still  know  little 
about  cost  growth,  and  we’re  still  confused  over  what  causes  it  (Drezner  and 
others,  1993;  Wilson,  1992:8).  This  is  significant  because  of  the  pervasive 
nature  of  cost  growth.  While  the  Department  of  Defense  can  run  stable 
programs  it  is  the  exception  not  the  norm  (Gansler,  1989:126).  In  other  words, 
cost  overruns  are  expected.  Eugene  Scott’s  opening  remarks  of  a  National 
Contract  Management  Journal  article  are  summarily  appropriate: 

Probably  the  most  troublesome,  difficult,  and  obstinate  problem  in  the 
execution  of  major  projects  is  the  phenomenon  of  cost  growth.  (Scott, 
1983:1) 

The  objective  of  this  research  is  to  determine  if  the  policies  implemented 
as  a  result  of  the  1986  Packard  Commission  Study  have  been  effective  in 
reducing  cost  overruns.  This  chapter  summarizes  a  number  of  published  works 
and  addresses  them  in  a  manner  which  seeks  to  not  only  prepare  the  reader  for 
resolving  the  overall  research  objective  but  to  also  clarify  some  of  the 
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misunderstandings  that  have  surrounded  the  study  of  cost  growth.  The  literature 
relates  to  the  following  areas  and  will  be  discussed  in  the  order  listed: 

•  The  importance  of  cost  grovi/th  studies  to  the  Department  of  Defense 

•  Factors  affecting  cost  growth. 

•  The  measurement  of  cost  growth. 

•  The  Department  of  Defense’s  response  to  cost  growth. 

•  The  Packard  Commission  study. 

The  Importance  of  Cost  Growth  to  the  Department  of  Defense  fPoD) 

Before  one  can  begin  to  address  the  issue  of  cost  growth  a  common 
definition  of  the  variable  to  be  measured  must  first  be  established.  Much  of  the 
current  literature  in  this  area  uses  the  terms  “cost  growth”  and  “cost  overruns” 
interchangeably  and  neglects  to  differentiate  between  the  two.  However,  the 
terms  are,  in  fact,  different,  and  this  paper  attempts  to  distinguish  between  the 
two  terms  and  will  discuss  that  difference  in  detail  later  in  this  chapter.  Simply 
speaking,  cost  growth  is  measured  as  the  difference  between  the  actual  costs  (or 
the  most  current  estimate  of  actual  costs)  and  the  initial  estimate  established  at 
the  start  of  a  system’s  development  (AFSC,  1988:A-22).  A  cost  overrun  is 
measured  as  the  difference  between  the  total  budget  for  all  the  work  on  the 
contract  and  the  estimated  final  cost  of  the  contract  (Christensen,  1993:44). 
Whether  one  is  talking  about  cost  growth  or  cost  overruns  the  underlying  notion 
is  the  same.  The  phenomenon  refers  to  the  tendency  for  the  unit  cost  of  a 
system  to  increase  during  the  course  of  the  acquisition  process  (Singer,  1983:2). 
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This  cost  increase  is  the  focus  of  this  thesis  and  has  been  the  focus  of  numerous 
studies  over  the  years.  At  least  ten  AFIT  theses  covering  cost  growth  research 
have  been  written  in  the  last  decade  and  for  more  than  25  years  RAND  has 
studied  topics  such  as  cost  growth  estimation  and  cost  control  and  has  analyzed 
almost  every  type  of  military  weapon  and  support  system  except  Navy  ships. 

The  majority  of  these  cost  growth  research  studies  reference 
Cost/Schedule  management  and  Cost/Schedule  Control  Systems  Criteria 
(C/SCSC)  as  the  tools  used  to  analyze  cost  and  performance  data  and  to 
provide  visibility  into  cost  and  schedule  progress.  Under  acquisition  reform  this 
system  has  evolved  into  what  is  now  called  “Earned  Value  Management  (EVM)”. 
“Earned  Value  Management  emphasizes  the  integration  of  the  cost,  schedule, 
and  technical  to  support  decisions  by  program  managers  ”  (ACQ  201, 

1997:251).  This  integration  then  allows  program  managers  to  evaluate  programs 
more  thoroughly  and  objectively.  The  EVM  System  is,  in  essence,  a  broader 
application  of  the  former  Cost/Schedule  Management  System,  and  it,  like 
C/SCSC,  establishes  a  set  of  standards  for  effective  management  systems. 
These  standards  constitute  a  fairly  rigorous  test  of  a  contractor’s  management 
techniques,  and  the  DoD  requires  contractors  reporting  cost  and  performance 
data  to  conform  to  them.  One  key  aspect  of  this  system  is  that  “contractors  use 
their  own  internal  integrated  management  system  to  manage  and  report  to  the 
government”  (ACQ  201 ,  1 997:254).  If  this  type  of  criteria  compliant  system  is 
used  properly  “it  ensures  that  valid  cost,  schedule  and  technical  progress 
information  provide  the  manager  with  an  effective  tool  for  decision  making” 
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(AFMCP  173-5,  1997).  The  purpose  of  the  system  is  to  standardize 
performance  data  reporting,  to  provide  the  contractor  and  the  Government 
Program  Managers  with  accurate  and  reliable  data  to  monitor  execution  of  their 
program,  and  to  provide  a  basis  for  responsible  decision  making  (DODI  5000.2, 
1996:Part  3.9). 

Undoubtedly,  cost  growth  is  a  popular  topic  and  it’s  easy  to  see  why.  In 
the  mid  1 980’s  perceived  inefficiencies  in  the  DoD’s  procurement  system  were  at 
their  peak  (Griffiths,  1985;  Gates,  1989).  Stories  of  the  Pentagon  paying  $400 
for  an  $8  claw  hammer  shoved  these  inefficiencies  into  the  spotlight  as  never 
before.  On  an  average  workday  in  the  1980’s  the  Pentagon  wrote  approximately 
52,000  contracts  which  works  out  to  about  15  million  contracts  each  year 
(Griffiths,  1985:144;  Gansler,  1989:4).  Furthermore,  the  DoD  was  spending 
around  $300  billion  a  year  (Gansler,  1989:4).  During  this  period  when  public 
pressure  on  controlling  defense  spending  was  at  its  peak  critics  estimated  that  in 
one  year  alone  sloppiness,  duplication,  and  fraud  could  add  up  to  $23  billion 
which  was  about  one  third  of  total  weapon  procurement  expenditures  (Griffiths, 
1985:144).  From  FY  1991-1996  the  DoD  spent,  on  average,  $135  billion  on 
procurement  and  wrote  approximately  nine  million  contracts  a  year  (DIOR, 

1997).  While  these  levels  are  considerably  lower  than  those  in  the  1980’s  the 
magnitude  of  DoD  procurement  activities  remains  significant,  and  over  the  last 
two  decades  it  seems  one  is  constantly  hearing  though  the  media  that  some 
defense  program  will  cost  taxpayers  millions  of  dollars  more  than  was  budgeted. 
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These  unplanned  cost  increases  are  of  interest  to  both  public  and  private 
organizations  (Gaston,  1992:Part  ll;Gansler,  1989:Ch  1).  A  1993  Rand 
Corporation  study  discussed  the  extent  of  cost  grovA^h  in  the  DoD.  Those  results 
indicated  that  cost  growth  has  fluctuated  around  20  percent  since  the  mid  1960’s 
and  that  little  improvement  has  occurred  overtime  (Drezner  and  others,  1993:2). 
Other  research  indicates  that  the  average  cost  overrun  on  DoD  acquisition 
contracts  is  approximately  40  percent  (Gansler,  1989:4).  However  one 
measures  the  unplanned  cost  increases  (growth  or  overruns)  is  academic;  the 
magnitude  of  the  problem  persists  and  is  readily  seen  as  20  to  40  percent  can 
represent  a  notable  loss.  For  example,  a  significant  contract,  as  defined  by 
Earned  Value  Management  Systems  Criteria  (EVMSC),  will  be  at  least  $70 
million  (in  constant  FY’96  dollars)  for  a  research,  development,  test,  and 
evaluation  contracts.  Significant  procurement  contracts  are  those  with  estimated 
values  of  at  least  $300  million  (FY’96  constant  dollars)(DoDI  5000.2, 1996:Part 
3,9-10).  Table  1  displays  the  enormity  of  cost  growth  or  overruns  on  $70  and 
$300  million  contracts  across  varying  growth  rates.  The  effect  of  a  10  to  50 
percent  cost  overrun  could  easily  range  from  $7  million  to  $150  million,  and  this 
represents  the  low  end  of  the  spectrum  as  these  contracts  just  satisfy  the 
significance  threshold  established  in  DoDI  5000.2. 
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Table  1.  Cost  Growth  Across  Varying  Growth  Rates 


1  %  Increase 

$70  Million 

$300  Million 

10% 

$  7  million 

$30  million 

$10.5  million 

$45  million 

$14  million 

$60  million 

25% 

$17.5  million 

$75  million 

30% 

$21  million 

$90  million 

35% 

$24.5  million 

$105  million 

40% 

$28  million 

$120  million 

45% 

$31.5  million 

$135  million 

50% 

$35  million 

$150  million 

Thus,  the  significance  of  the  cost  impact  is  apparent,  but  the  issue  of  cost 
is  not  the  only  concern  to  the  DoD.  “Cost  growth  forces  the  DoD  to  revise 
budget  plans,  makes  systems  less  affordable,  and  frequently  erodes 
congressional  support  for  acquisition  programs”  (Tyson  and  others,  1994:1-1). 
Furthermore,  it  affects  the  quality  of  decisions  concerning  U.S.  defense  policy 
(Drezner  and  others,  1993:1).  It  is  also  important  to  the  DoD  for  the  following 
reasons: 

-  It  affects  the  fielding  of  a  system. 

-  It  affects  the  defense  industrial  base. 

-  It  affects  the  common  welfare  of  the  people. 


Cost  Growth  Affects  the  Fielding  of  a  System.  Unplanned  increases  in 
cost  have  to  be  offset  somewhere.  One  of  these  offsets  is  often  reflected  in  the 
schedule.  Because  of  inadequate  funding  schedules  often  have  to  be  slipped 
(Marciniak,  1990:213).  This  view  is  also  mentioned  by  Gansler  who  states  that 
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"as  increasing  costs  confront  a  fixed  or  declining  budget  the  only  way  to  fit  the 
higher  costs  into  the  budget  is  to  stretch  out  the  program,  by  extending  its 
development  time  and/or  by  buying  fewer  production  units  each  year”(Gansler, 
1989:121).  There  don’t  appear  to  be  a  number  of  options  available  to  offset  the 
unplanned  costs,  and  the  argument  is  fairly  simple.  As  costs  escalate  the  DoD 
offsets  the  increase  by  stretching  out  schedules  or  by  buying  lesser  quantities. 
Both  of  these  responses  affect  the  fielding  of  the  system  and  contravene  DoD 
acquisition  policy.  “The  primary  objective  of  the  defense  acquisition  system  is  to 
acquire  quality  products  that  satisfy  the  needs  of  the  operational  user  with 
measurable  improvements  to  mission  accomplishment,  in  a  timely  manner,  at  a 
fair  and  reasonable  price”  (DODI  5000.1,  1996:3).  Cost  growth  hinders  the 
attainment  of  that  objective  by  delaying  the  provision,  thereby  leaving  U.S.  forces 
without  needed  capabilities,  or  by  providing  lesser  quantities  than  what  is 
required  in  the  field.  As  Gansler  states,  “..overall  military  effectiveness  is  not 
measured  only  in  terms  of  a  single  weapon’s  performance,  quantities  matter,  too” 
(Gansler,  1989:172).  He  also  states  that  “many  military  theorists  argue  that 
quantity  has  an  even  greater  effect  than  individual  weapon  performance” 
(Gansler,  1989:172). 

Cost  Growth  Affects  the  Defense  Industrial  Base.  The  reduced 
quantities  produced  as  an  offset  for  cost  overruns  can  also  affect  the  defense 
industrial  base.  Over  the  years  there  has  been  a  slow  decline  in  the  number  of 
defense  oriented  firms,  and  this  shrinking  is  warranted  because  of  the  reduction 
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in  the  total  volume  being  produced  (Gansler,  1989:256).  However,  this  smaller 
defense  industrial  base  could  affect  overall  efficiency  and  could  also  increase  our 
country’s  dependency  on  foreign  suppliers  which  could  place  the  United  States 
in  a  difficult  position  in  times  of  war.  Since  World  War  il  “there  has  always  been 
a  concern  about  the  availability”  and  the  dependability  of  obtaining  materials 
from  foreign  sources,  and  a  shrunken  defense  industrial  base  may  find  it  difficult 
or  impossible  to  timely  provide  sufficient  resources  in  a  period  of  crisis  or  conflict 
(Gansler,  1989:266;  Weida,  1987:113). 

Cost  Growth  Affects  the  Common  Welfare  of  the  People.  One  of  the 
most  difficult  aspects  of  budgeting  is  establishing  priorities.  Our  nation’s  people, 
as  a  whole,  have  an  indirect  voice  in  how  the  country’s  priorities  are  established 
by  voting  for  elected  officials  they  feel  best  represent  their  desires  for  certain 
public  goods  such  as  national  defense  or  education.  Cost  overruns  result  in  only 
two  options:  either  funding  is  drawn  away  from  other  public  goods  in  order  to 
fund  the  defense  good  or  the  defense  good  must  be  foregone.  In  either  case  the 
public’s  needs  may  not  be  satisfied.  Public  opinion  on  the  costs  of  defense  has 
always  been  tense.  Consider  President  Eisenhower’s  remarks  in  April  1953 
concerning  the  futility  of  military  spending. 

-’’Every  gun  that  is  made,  every  warship  launched,  every  rocket  fired 
signifies,  in  the  final  sense,  a  theft  from  those  who  hunger  and  are  not  fed, 
those  who  are  cold  and  are  not  clothed... 

-The  cost  of  one  modem  heavy  bomber  is  this:  a  modem  brick  school  in 
more  than  30  cities. 

-It  is  two  eiectric  power  plants,  each  serving  a  town  of  60,000. 
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It  is  two  fine,  fully  equipped  hospitals. 

-It  is  some  50  miles  of  concrete  highway. 

-We  pay  for  a  single  fighter  plane  with  a  half  million  bushels  of  wheat 

-We  pay  for  a  single  destroyer  with  new  homes  that  could  have  housed 

more  than  8,000 people.”  (Gaston,  1992:91-93) 

Gansler  refers  to  this  debate  as  a  “guns  versus  butter”  debate,  and  states 
that  the  issue  has  “become  a  dominant  issue  during  periods  of  peace  since  the 
end  of  the  Korean  war”  (Gansler,  1989:79).  This  line  of  public  thought  considers 
heavily  the  tradeoff  between  defense  goods  and  other  public  goods.  Because 
this  competition  exists  between  the  two  one  needs  to  consider  that  people  are 
only  willing  to  sacrifice  so  much  of  the  national  output  for  defense  spending 
(Peterson,  1988:  702).  Cost  overruns  upset  the  “balance”  of  public  opinion.  At 
some  point  the  public,  through  their  elected  officials,  will  refuse  to  support 
programs  experiencing  cost  overruns.  These  programs,  as  public  goods,  may 
be  canceled. 

These  unplanned  cost  increases  could  easily  add  up  to  hundreds  of 
millions  of  dollars,  but  perhaps  more  important  are  the  consequences  cost 
growth  may  force  on  the  operational  user  and  on  this  country’s  citizenry.  The 
magnitude  of  this  procurement  phenomenon  may  signal  cost  growth  as  one  of 
the  acquisition  system’s  chief  concerns.  It’s  certainly  a  significant  problem  in 
tight  budget  environments  (Drezner  and  others,  1993:xi).  Identifying  possibly 
factors  affecting  cost  growth  may  lead  decision  makers  to  a  better  understanding 
of  the  problem  and  may  provide  them  with  improved  remedies  for  mitigating  its 
effects. 
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Factors  Affecting  Cost  Growth  in  the  Department  of  Defense 

When  the  public  considers  the  reasons  for  cost  growth  normally  the  first 
impression  that  comes  to  mind  is  that  “cost  growth  is  the  result  of  overruns 
because  of  gross  mismanagement”  (Scott,  1983:38).  While  it  is  true  that 
mismanagement  or  other  inefficiencies  are  sometimes  involved  most  often  these 
factors  are  responsible  for  only  a  relatively  small  portion  of  the  cost  growth 
(Dreznerand  others,  1993:xi-xv;  Scott,  1983:37-38;  Weida,  1987:Ch  9).  It’s 
difficult  to  understand  all  of  the  factors  which  lead  to  increased  costs,  and, 
unfortunately,  only  the  widely  publicized  and  readily  grasped  factors  such  as 
fraud  or  mismanagement  tend  to  get  undue  attention  (Weida,  1987:  Ch  8; 
Griffiths,  1985),  Actually,  the  minor  sins  of  fraud,  waste,  and  abuse  “cost  the 
taxpayer  only  a  fraction  of  the  amount  generated  by  other  inefficiencies”  (Weida, 
1987:145).  Table  2  shows  a  number  of  possible  factors  identified  as  affecting 
cost  growth  (Scott,  1983).  Independently  these  factors  may  cause  cost  growth, 
but  usually  one  or  more  (or  a  combination  of  many)  are  the  real  causes  (Scott, 
1983:38). 
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Table  2.  Factors  Affecting  Cost  Growth 


Planning  Difficulties 

Risk  Elements 

Management 

Inefficiencies 

1.  Incomplete  Definition 
of  Work 

6.  Unforeseeable 
Conditions 

1 1 .  Disorganized  Work 
Direction  and  Productivity 

2.  Interface 
Incompatibilities 

7.  Unpredictable 
Regulatory  or  Funding 
Delays 

12.  Subcontracting 

3.  Changes;  Failure  to 
Anticipate  Needs 

8.  Unforeseen  Technical 
Difficulties 

13.  Unnecessary  Work  or 
“Gold  Plating” 

4.  Estimating 

Uncertainties;  Poor 
Estimating 

9.  Uncontrollable  Forces 

14.  Project  Control 

5.  Optimistic 

Assumptions 

10.  Unanticipated 
Economic  Conditions 

1 5.  Work  Load 

Projections 

The  factors  represented  in  the  planning  difficulties  category  are  those  that 
tend  to  prevent  realistic  early  assessments  of  the  final  costs  of  projects.  For 
example,  budgets  and  estimates  are  established  during  the  planning  stages  of  a 
project  while  the  project  is  still  in  the  process  of  being  defined.  However, 
defining  a  project  can  be  “carried  initially  only  to  a  point  of  verifying  feasibility  and 
defining  basic  requirements,  but  must  stop  short  of  including  complete  detail” 
(Scott,  1983:38).  So,  contractors  often  find  themselves  in  a  “catch  22”  situation 
wherein  if  they  spend  too  much  time  defining  a  project  to  ensure  accurate 
estimation  overall  costs  will  rise  as  schedules  are  extended,  and  if  they  define 
projects  inadequately  or  incompletely  estimations  of  costs  are  difficult  to 
determine  as  rigorous  estimates  require  a  good  definition  of  the  work.  In  addition 
it’s  sometimes  impossible  to  project  future  requirements  even  for  a  few  years. 
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Political,  international,  and  cultural  forces,  what  Scott  calls  the  natural  evolution 
of  progress  make  changes  inevitable  and  unpredictable  (Scott,  1983:45). 

The  risk  element  factors  are  those  that  are  more  inherent  in  the  system; 
that  is,  they  are  not  controllable  nor  are  they  predictable.  Risk  factors  “are  those 
adverse  possibilities  which  cannot  be  reliably  anticipated.  These  are 
potentialities  that  may  occur  or  they  may  not,  and  there  is  no  reasonable  way  to 
take  appropriate  preventive  measures”  (Scott,  1983:41).  No  matter  how  good 
our  estimating  models  are  or  how  thorough  our  planning  process  is  conditions 
can  still  arise  that  can’t  be  reasonably  foreseen.  Technological  change  is  one 
condition  that  arises  that  is  often  unforeseeable  yet  tends  to  drive  costs  up. 

Many  decision  makers  believe  that  “the  best  way  to  start  and  to  defend 
successfully  a  weapon  development  program  is  to  employ  the  most  advanced 
technology  available.  Unfortunately,  this  is  also  the  best  way  to  guarantee  that 
the  costs  associated  with  that  program  will  be  high”  (Weida,  1987:152-153). 

The  last  category,  management  inefficiencies,  contains  those  factors 
mentioned  previously.  These  are  the  factors  that  are  considered  to  be 
controllable  by  management.  Any  cost  overruns  that  are  actually  caused  by 
mismanagement  fall  in  this  category.  Proper  managerial  action  may  avoid  some 
of  the  overruns  resulting  from  these  causes,  but  it’s  doubtful  that  all  of  it  could  be 
eradicated  as  management  inefficiencies  are  bound  to  exist  (Scott,  1983:38-45). 

This  breakdown  is  similar  to  that  identified  in  other  cost  growth  research. 
Similar  studies  have  identified  the  following  factors  as  other  potential  causes  of 
cost  grov\rth: 
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1 .  Not  adequately  budgeting  for  risk  and  uncertainty  (particularly  during 
the  relatively  high  risk  research  and  development  phase) 

2.  Unexpected  high  inflation 

3.  Supply  and  demand  factors 

4.  Poor  resource  allocation 

5.  Managerial  inefficiency 

6.  Technological  uncertainty 

7.  System  requirements  uncertainty 

8.  Less  than  perfect  cost  estimating  techniques 

9.  The  budget  process  involving  risk  in  congressional  appropriations. 

10.  Program  length,  size,  and  maturity  (Dreznerand  others,  1993:xiii) 
(Woodward,  1983:13). 

It’s  natural  to  question  the  efficiency  of  the  acquisition  process  because 
the  DoD’s  procurement  budget  is  so  large.  As  stated  previously,  the  DoD  spent 
upwards  of  $300  billion  a  year  on  various  programs  in  the  1980’s  and  spends 
close  to  $140  billion  a  year  in  the  1990’s.  We’ve  also  demonstrated  where  even 
a  small  inefficiency  could  translate  into  substantial  amounts  of  wasted  funds. 
Unfortunately,  there  aren’t  any  indices  that  directly  measure  procurement 
inefficiencies,  so  analysts  are  forced  to  use  quantitative  measures  of  program 
outcomes,  such  as  cost  growth,  as  proxies  for  inefficiency  (Gates,  1989:9).  Over 
time,  cost  growth,  as  a  proxy,  has  come  to  be  viewed  as  a  program  inefficiency, 
and  reforms  have  addressed  it  rather  than  the  underlying  causes 
(Gates,  1989:9).  However,  cost  growth  may  not  signal  an  inefficiency  in  the 
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program.  For  example,  if  a  program  is  managed  efficiently  but  the  initial  estimate 
was  low  then  cost  growth  isn’t  necessarily  inefficient,  and  if  the  initial  estimate 
was  excessive  than  the  absence  of  cost  growth  may  not  indicate  program 
efficiency.  It’s  important  to  remember  that  cost  growth  is  only  a  proxy  which 
makes  it  difficult  to  determine  if  program  inefficiencies  are  truly  significant,  and 
any  reform  efforts  the  DoD  implements  need  to  fix  the  problem  not  just  the 
symptoms. 

As  stated  earlier,  cost  growth  could  result  from  mismanagement,  fraud, 
waste,  or  abuse  which  is  consistent  with  the  view  that  cost  grov\/th  is  a  direct 
indication  of  program  inefficiencies.  Gates  reports  that  the  appropriate 
response,  in  this  case,  “is  to  improve  DoD’s  cost  management  capabilities” 
(Gates,  1989:10).  This  would  include  measures  such  as  improving  cost  data, 
cost  estimating  techniques,  and  the  qualifications  of  DoD’s  program  managers. 
Reform  efforts  containing  these  measures  may  be  successful  if  cost  growth  is 
the  result  of  mismanagement  and  fraud,  but  they  won’t  work  if  cost  growth  is 
merely  a  signal  of  other  problems  (Gates,  1989:9-1 1). 

Also,  the  impact  of  an  organization’s  culture  on  cost  growth  can  not  be 
ignored.  Organizational  culture  refers  to  the  “broader  values  and  normative 
patterns  which  guide  worker  behavior,  practices  and  policies”,  and  plays  a 
crucial  role  as  a  mechanism  of  control  (  Flamholtz,  1983:158,168).  An  example 
of  how  culture  affects  cost  growth  can  be  seen  in  the  case  of  the  Navy’s  A-12 
program.  The  information  the  program  manager  reported  up  the  chain  of 
command  was  always  cast  in  a  positive,  optimistic  light.  Rather  than  highlighting 


21 


risks  “in  a  system  which  depends  upon  information  being  pushed  up  from  the 

bottom,”  he  viewed  unfavorable  variances  through  “rose  colored”  glasses  and 

chose  to  continually  report  how  “things  would  get  better”  (Beach,  1990:41). 

Unfortunately,  this  is  a  by-product  of  the  control  system  (  Flamholtz,  1983:168). 

The  same  performance  measurements  use  to  evaluate  the  system  are  also  used 

to  evaluate  the  performance  of  individuals  and  often  results  in  a  “shoot  the 

messenger”  mentality.  Managers  may  then  make  the  more  parochial,  short  term 

decisions,  or  in  the  case  of  the  A-12  Program  Manager,  may  continually  report 

overly  optimistic  information  for  fear  of  poor  evaluations.  Note,  however,  this 

type  of  behavior  is  not  unique  to  the  Navy  and  can  be  generalized  to  other 

services.  In  the  Navy  A-12  inquiry  report  the  inquiry  officer  concluded,  in  part: 

There  is  no  reason  to  believe  that  the  factors  which  made  these  officials 
[the  A-12  managers]  choose  to  respond  the  way  they  did  are  unique  to 
this  Military  Department.  Indeed,  experience  suggests  that  they  are  not. 
Unless  means  can  be  found  to  solve  this  abiding  cultural  problem,  the 
failures  evidenced  in  this  report  can  be  anticipated  to  occur  again  in  the 
same  or  a  similar  form.  (Beach  report,  1990:41) 

Another  possible  cause  of  cost  growth  not  already  discussed  concerns  the 
initial  estimation  process.  Inadvertent  or  intentional  underestimation  of  initial 
costs  will  lead  to  cost  growth.  The  first  explanation  has  often  been  referred  to  as 
the  “winners  curse”  (Quirk  and  Terasawa,  1984).  At  the  beginning  of  an 
acquisition  program  the  development  and  procurement  costs  are  uncertain,  yet 
contractors  are  forced  to  estimate  their  expected  costs  which  are  then  used  in 
the  selection  process.  If  all  of  the  other  evaluation  factors  are  equal  the 
contractor  selected  will  be  the  one  who  underestimates  his  cost  the  most. 


22 


Underestimation  in  this  case  is  caused  by  uncertainty  and  is  entirely  inadvertent, 
yet  “this  introduces  a  selection  bias  toward  programs  with  a  predisposition  for 
cost  growth,  even  though  the  program  may  be  conducted  efficiently”  (Gates, 
1989:10).  Gansler  also  reports  that  “the  system  encourages  a  great  deal  of 
“optimism”  in  bidding  and  in  budgeting”  (Gansler,  1989:177).  The  winning 
contractor  is  one  who,  all  other  factors  being  equal,  bids  the  lowest  amount. 

Cost  growth,  in  this  case,  doesn’t  signal  an  inefficiency  in  the  program,  but  it  may 
signal  that  there  are  inefficiencies  in  the  selection  process  (Gates,  1989:10). 

Any  reform  measures  attempting  to  improve  cost  management  capabilities  in  this 
situation  would  not  be  effective.  The  second  explanation  results  from 
competition  and  is  often  referred  to  as  “buying  in”.  Competitive  pressures 
encourage  contractors  to  make  overly  optimistic  cost  estimations  during  the 
proposal  process.  Gansler  points  out  that  the  evaluators  know  that  awards  will 
be  “subjected  to  enormous  public  scrutiny,  and  will  attempt  to  take  the  safe  route 
by  making  the  award  to  the  contractor  who  promises  the  most  and  offers  the 
lowest  price”  (Gansler,  1989:179-180).  There  is  an  intense  pressure  on  the 
contractors  to  compete  vigorously  as  very  few  major  weapon  system  programs 
are  initiated  in  a  decade,  for  example.  In  this  case  initial  cost  underestimation  is 
intentional  rather  than  inadvertent,  and  any  reform  measures  appropriate  in 
these  instances  must  be  different  than  in  the  earlier  situations  in  order  to  be 
effective  (Gates,  1989:9-13). 

Program  turbulence  is  often  thought  of  as  another  cause  of  cost  growth 
(Dreznerand  others,  1993;  Gansler,  1989:Ch  6,  Gates,  1989).  Program 
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turbulence  is  a  term  used  to  describe  changes  or  modifications  made  to  a 
program  such  as  production  rate  changes,  schedule  changes,  or  changes  in 
end-items  required  (Gates,  1989: 10-11).  However,  Gates  points  out  that 
program  turbulence  is  actually  a  symptom  of  other  problems  and  not  a  cause  of 
cost  growth.  He  reports  that  unanticipated  inflation  and  increases  in  the 
contractor’s  unit  cost  estimates  lead  to  funding  shortfalls,  and  that  these 
shortfalls  cause  production  schedule  changes.  Furthermore,  unexpected 
technical  problems,  changes  in  the  total  procurement  objective,  and  program 
funding  changes  also  lead  to  changes  in  the  production  schedule  (Gates, 
1989:11).  These  changes,  however,  may  not  be  bad.  “In  fact,  program 
turbulence  may  be  an  efficient  response  to  unavoidable  and  unpredictable 
changes  in  national  priorities,  the  defense  environment,  or  the  program  itself 
(Gates,  1989:11).  The  impact  of  changes  must  also  be  considered.  OneAFIT 
study  tested  the  popular  assumption  that  changes  on  defense  contracts  are 
correlated  with  poor  cost  performance.  However,  the  results  of  that  study  did  not 
support  that  assertion.  The  results  failed  to  show  evidence  that  the 
hypothesized  relationship  between  changes  and  cost  performance  existed 
(Gordon,  1996).  The  bottom  line  is  that  change  may  not  be  bad.  Together, 
these  studies  lead  us  to  believe  that: 

1)  program  turbulence  (changes)  may  not  lead  to  higher  cost  growth,  and 

2)  program  turbulence  may  be  an  efficient  response  to  unavoidable  and 

unpredictable  change. 
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Therefore,  if  this  is  the  case,  any  reform  measures  attempting  to  remove 
program  turbulence  or  stabilize  high  priority  programs  should  be  carefully 
viewed.  Any  interventive  efforts  may  simply  “shift  the  burden  of  adjustment  to 
unprotected  programs”,  and  the  added  costs  of  turbulence  in  these  programs 
needs  to  be  considered  (Gates,  1989:11). 

The  factors  causing  the  cost  growth  problem  need  to  be,  first,  identified  in 
order  to  control  or  solve  the  problem.  Once  these  causal  factors  have  been 
determined  steps  can  then  be  taken  to  implement  policies  and  other  reform 
measures  aimed  at  eliminating,  reducing,  or  stabilizing  the  causes.  One  needs 
to  remember,  though,  that  cost  growth  is  only  a  proxy  which  makes  it  difficult  to 
determine  if  program  inefficiencies  are  truly  significant,  and  any  reform  efforts  the 
DoD  implements  need  to  address  and  fix  the  underlying  problems  not  just  the 
symptoms. 

The  Measurement  of  Cost  Growth 

Cost  gro\wth  can  be  measured  in  a  number  of  different  ways,  and  literature 
presents  different  uses  of  the  term.  One  runs  across  the  terms  cost  growth,  cost 
variance,  and  cost  overruns  when  reviewing  studies  in  this  area.  It  seems  the 
terms  “cost  growth”  and  “cost  overrun”  are  often  used  interchangeably,  but  they 
are  in  fact  referring  to  different  measurements.  This  may  lead  to 
misunderstanding  and  confusion  among  policy  makers,  program  managers,  and 
contractors  who  are  unaware  of  the  difference.  Furthermore,  it  makes  it  difficult 
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to  compare  available  research  work  and  studies  because  the  term  is  not 
standardized  (Sapp,  1971:15). 

Cost  Growth.  “Cost  growth  can  be  defined  simplistically  as  the  difference 
between  estimated  and  actual  costs”  (Drezner  and  others,  1993:1).  As  defined 
in  the  AFSC  Cost  Estimating  Handbook,  cost  growth  is  measured  as  the 
difference  between  the  actual  costs  (or  the  most  current  estimate  of  actual  costs) 
and  the  initial  estimate  established  at  the  start  of  a  system’s  development.  It  is  a 
term  “related  to  the  net  change  of  an  estimate  or  actual  amount  over  a  base  cost 
figure  previously  established”  (AFSC,  1988:A-22).  Therefore,  the  total  cost 
growth  of  a  contract  is  simply  calculated  as  shown  in  Equation  1  by  subtracting 
the  initial  cost  estimate  from  the  estimate  at  completion. 

Cost  _  _  initial  (1) 

Growth  estimate 

Cost  Overruns.  The  terms  cost  overrun  and  cost  variance  refer  to  same 
measurement  (Christensen,  1993:44).  Specifically,  a  cost  overrun  is  simply  an 
adverse  cost  variance.  The  formula  for  calculating  cost  variance  is  found  in  Air 
Force  Systems  Command  Pamphlet  (AFSCP)  173-4,  Guide  to  Analysis  of 
Contractor  Cost  Data  and  is  calculated  by  subtracting  the  actual  cost  of  work 
performed  (ACWP)  from  the  budgeted  cost  of  work  performed  (BCWP)  (AFSCP, 
1989).  Christensen  states  the  formula  differently  but  achieves  the  same  results. 
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He  writes  that  “the  overrun  at  completion  is  the  difference  between  the  total 
budget  for  all  the  work  on  the  contract,  termed  the  “Budget  At  Completion”  (BAC) 
and  the  estimated  final  cost  of  the  contract,  termed  the  “Estimate  At  Completion” 
(EAC)  (Christensen,  1993:44).  This  is  the  same  formula  as  the  BCWP  of  a 
completed  contract  equals  the  BAC,  and  the  ACWP  of  a  completed  contract 
equals  the  EAC.  This  is  demonstrated  below  in  Equation  2. 

Cost  ^  BCWP  ACWP  . 

Overrun  (BAC)  '  (EAC)  '  ’ 

Cost  Growth  Versus  Cost  Overruns.  It’s  important  that  one 
understands  the  difference  between  the  two  measures.  Figure  1  provides  a 
graphic  description  of  cost  growth  as  we  now  understand  it.  The  cost  growth  of  a 
completed  contract  will  capture  the  unplanned  cost  increases  occurring  as  a 
result  of  low  estimations,  changes  to  the  contract,  and  any  management 
inefficiencies.  Figure  2  shows  a  similar  model  for  cost  overruns.  The  key  feature 
of  the  cost  overrun  model  is  the  use  of  the  current  budget  as  opposed  to  the 
initial  estimate  used  in  the  cost  growth  model.  As  a  contract  progresses  budgets 
are  revised  and  updated,  and  these  “current  budgets”  are  then  recorded  as  the 
BCWP.  As  a  contract  progresses  any  costs  from  changes  and  modifications  are, 
therefore,  absorbed  in  these  updated  budgets;  thus,  a  cost  overrun  would  only 
reflect  increases  caused  by  low  estimations  or  possibly  mismanagement  of  some 
sort.  We  understand  from  Gates’  research  that  changes  (or  turbulence)  in  the 
contract  is  not  necessarily  inefficient,  and  Scott  leads  us  to  believe  that  a  number 
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of  changes  are  due  to  unforeseeable  risk  elements  which  cannot  be  planned  for 
anyway.  Furthermore,  the  conclusions  drawn  from  Gordon’s  thesis  suggest  that 
modifications  don’t  appear  to  be  highly  correlated  with  cost  growth.  For  these 
reasons  it  is  my  opinion  that  the  use  of  the  cost  overrun  model  is  more 
appropriate  as  a  measurement  of  unplanned  cost  increases.  This  thesis 
recognizes  the  contributions  made  by  both  models,  but  the  cost  overrun  model 
will  be  used  as  the  measuring  tool  throughout  this  study. 
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The  Department  of  Defense’s  Response  to  Cost  Growth 


As  stated  previously,  for  several  years  the  Department  of  Defense  has 
recognized  that  cost  growth  in  weapons  systems  is  a  problem.  In  fact, 
regulations  to  control  defense  procurements  extend  as  far  back  as  the  1940’s. 
The  Armed  Services  Procurement  Act  of  1947  was  essentially  the  first  formal 
unified  defense  procurement  policy  to  be  established  (Gates,  1989:2;  Gansler, 
1989:179).  Early  procurement  reform  efforts  in  the  DoD  focused  on  coordinating 
procurement  reform  among  the  services.  The  various  service’s  mission  were 
ambiguous,  inter-service  competition  was  high,  and  in  a  number  of  areas 
weapons  programs  overlapped  (Acker,  1980;  Gates,  1989).  Drezner  discussed 
cost  growth  trends  in  a  Rand  study  entitled.  An  Analysis  of  Weapon  System  Cost 
Growth,  and  showed  where  several  attempts  have  been  made  over  the  years  to 
improve  cost  performance.  Table  3  is  an  excerpt  from  that  study  and  shows 
some  of  the  more  important  regulatory  and  administrative  initiatives  implemented 
over  the  last  25  years  that  were  intended  to  improve  cost  performance  in  weapon 
system  development  (Drezner  and  others,  1993:28-29).  Some  of  the  more 
recent  regulatory  and  administrative  initiatives  have  been  added  to  Drezner’s  list. 
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Table  3.  Acquisition  Regulations  &  Initiatives  Over  The  Years 


I  Year 

Regulations  1  Initiatives 

1969 

Packard  initiatives  published 

1970 

1971 

Blue  Ribbon  Defense  Panel  (Fitzhugh  Commission) 

1972 

DoDD  5000.1  (Major  System  Acquisitions); 
Commission  on  Govt  Procurement 

1973 

DoDD  5000.4  (CAIG);  DoDD  5000.3  (T&E) 

1974 

1975 

DoDI  5000.2  (Major  System  Acquisitions);  DoDD 
5000.28  (DTC) 

1976 

0MB  Circular  A-1 09 

1977 

1978 

Acquisition  Cycle  task  Force 

1979 

Defense  Resource  Mngt  Study 

1980 

1981 

Carlucci  Initiatives  (AlP) 

1982 

Nunn-McCurdy  (thresholds) 

1983 

Grace  Commission 

1984 

CICA 

1985 

DoDD  5000.43  (streamlining) 

1986 

Packard  Commission 

1987 

DoDD  5134.1  (USD(A));  DoDD  5000.49  (DAB) 

1988 

1989 

Defense  Management  Review 

1990 

1991 

Revised  DoDI  5000.2  (Major  System  Acquisitions) 

1992 

1993 

1994 

Federal  Acquisition  Streamlining  Act  (FASA) 

1995 

Federal  Acquisition  Improvement  Act  (nicknamed 
FASA  II) 

1996 

0MB  Circular  A-76 

In  the  1960’s,  Under  Secretary  of  Defense  McNamara’s  direction, 
quantitative  measures  of  program  performance  became  more  important  and 
showed  that  cost  growth  was  becoming  increasingly  significant  (Gates,  1989:3-4; 
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Enthoven,  1971).  As  a  result,  from  1961-1969,  McNamara  introduced  a  number 
of  initiatives  aimed  at  reducing  and  controlling  cost  grovvth.  However,  cost 
grov\/th,  perceived  as  an  inefficiency  in  the  procurement  process,  continued 
through  the  1970’s,  becoming  more  and  more  prevalent.  At  that  time  it  was 
generally  acknowledged  to  be  a  problem  in  the  DoD.  Excessive  competition  was 
one  of  the  most  frequently  cited  causes  for  problems  such  as  cost  growth  and 
schedule  delays  (Gates,  1989:4).  David  Packard,  Deputy  Secretary  of  Defense 
(DEPSECDEF)  who  took  office  in  1969  along  with  Secretary  of  Defense  Melvin 
Laird,  published  a  set  of  initiatives  which  had  considerable  impact  on  acquisition 
policy  (Dews,  1979:1).  These  initiatives  were  believed  to  have  had  a  profound 
effect  on  cost  growth.  In  fact,  a  1979  Rand  Corporation  study.  Acquisition  Policy 
Effectiveness:  Department  of  Defense  Experience  in  the  1970’s  showed  that 
cost  gro\A/th  for  major  weapon  systems  averaged  approximately  8%  in  the  1960’s 
but  decreased  to  approximately  7%  in  the  1970’s  (Dews,  1979:56). 

However,  “when  Frank  Carlucci  became  the  DEPSECDEF  in  1980, 
alleged  acquisition  inefficiencies  (cost  growth,  schedule  delays,  and  performance 
shortfalls)  continued  to  plague  the  defense  acquisition  process”  and  Congress 
and  the  General  Accounting  Office  were  becoming  increasingly  concerned  about 
cost  growth  (Gates,  1989:6).  What  was  thought  to  have  been  controlled  in  the 
1970’s  (cost  grov/th)  resurfaced  as  a  pressing  problem  in  the  1980’s.  In  fact, 
cost  growth  in  the  1980’s  seemed  to  be  increasing  (Singer,  1983:2;  Drezner  and 
others,  1993:30).  Looking  back  at  the  alleged  success  of  reduced  cost  growth  in 
the  1970’s  Drezner  found  that  the  differences  in  the  average  age  of  programs 
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largely  accounted  for  the  apparent  improvement  and  that  programs  tended  to 
incur  more  cost  growth  as  they  matured  because  of  accumulated  problems. 
“Taking  that  into  account,  it  appears  that,  on  average,  weighted  average  total 
program  cost  growth  has  been  fairly  constant  over  time,  averaging  around  20 
percent”  (Drezner  and  others,  1993:30).  It  appears  that  none  of  the  regulatory 
and  administrative  initiatives  implemented  in  the  60’s,  70’s,  or  early  80’s  were 
successful  in  reducing  cost  growth  in  major  weapon  system  contracts. 

The  Packard  Commission  Study 

The  Packard  Commission  was  established  by  President  Reagan  in  an 
effort  to  combat  the  still  pervasive  inefficiencies  in  the  DoD’s  procurement 
system  and  to  stimulate  procurement  reform.  In  the  mid  1980’s  cost  growth, 
schedule  delays,  performance  shortfalls,  and  high  unit  procurement  costs  were 
still  perceived  as  major  problems  in  the  acquisition  process  (Gates,  1989:6-8). 
The  group,  officially  called  the  President’s  Blue  Ribbon  Commission  on  Defense 
Management,  was  informally  named  after  its  chairman,  industrialist  and  former 
DEPSECDEF  David  Packard.  They  examined  DoD  management  in  general  but 
concentrated  particularly  on  the  acquisition  process. 

William  J.  Perry,  former  Under  Secretary  of  Defense  for  Research  and 
Engineering  and  a  member  of  the  Commission  stated  that,  “the  primary 
conclusion  of  the  Packard  Commission  was  that  defense  acquisition  was 


unacceptably  inefficient.  Specifically,  major  weapons  systems  cost  too  much 
take  too  long  to  field  and  by  the  time  they  are  fielded  incorporate  obsolete 


technology”  (Nordwall,  1987:80).  He  also  reported  the  problems  didn’t  appear  to 
be  the  result  of  deviations  from  or  abuses  of  the  system  but  were  direct 
consequences  of  the  system  itself.  A  GAO  report  authored  by  the  Subcommittee 
on  Defense  Industry  and  Technology  pointed  out  that  the  Commission  found  the 
DoD’s  acquisition  system  had  become  an  “increasingly  bureaucratic  and  over¬ 
regulated  process  in  which  acquisition  policy-making  and  program  management 
responsibility  were  fragmented  and  diluted”  (GAO,  1991:1-2).  The  Commission 
reported  that  the  acquisition  system  had  become  so  highly  competitive  that 
program  managers,  because  of  internal  and  external  pressures,  became 
“supplicants  of,  rather  than  managers  of,  major  new  defense  systems”  (GAO, 
1991:2).  Perry  commented  that  these  competitive  pressures  resulted  in  a 
“huckster”  environment  that  led  program  managers  to  optimistically  interpret 
information  about  a  system’s  cost,  schedule,  and  performance  (GAO,  1991; 
Nordwall,  1987:80-81).  These  remarks  echo  those  made  in  the  Beach  report 
regarding  the  organizational  culture  described  earlier  regarding  the  Navy  A-12 
program.  Because  of  this  culture  of  optimism  programs  inevitably  ended  up  with 
understated  costs  and  there  was  a  high  incident  of  cost  overruns  on  major 
weapon  systems  programs. 

The  Commission  concluded  that  the  primary  problems  with  the  acquisition 
process  were  the  same  inefficiencies  of  the  60’s  and  70’s,  namely  cost  growth, 
schedule  delays,  and  performance  shortfalls.  It’s  important  to  note  that  the 
Commission  believed  these  inefficiencies  were  actually  the  problems  and  not  just 
symptoms.  Recall  from  the  discussion  earlier  in  this  section  that  for  reform 
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efforts  to  be  effective  they  must  address  the  root  problem  not  just  fix  the 
symptoms.  This  seems  to  be  the  first  time  that  symptoms  and  problems  were 
differentiated.  They  also  believed  that  the  perceived  causes  of  these  problems 
were  program  instability,  lengthening  acquisition  schedules,  “gold  plating”,  and 
program  “buy-ins”  (Gates,  1989:7). 

The  Packard  Commission  offered  a  number  of  recommendations  to 
alleviate  these  problems.  Essentially,  the  Commission’s  report  “boiled  down”  to 
the  following  recommendations: 

•  streamlining  the  acquisition  process 

•  increasing  tests  and  prototyping 

•  changing  the  organizational  culture 

•  improving  planning 

•  adopting  the  competitive  firm  model  where  appropriate 
President  Reagan  endorsed  the  Packard  Commission’s  recommendations  in  the 
National  Security  Decision  Directive  219  in  1986  and  they  were  incorporated  in 
the  Defense  Reorganization  Act  (Gates,  1989:7;  GAO,  1990:1). 

The  Packard  Commission’s  study  did  not  appear  to  break  new  ground. 
Actually,  their  reform  recommendations  are  strikingly  similar  to  previous  reform 
efforts  (Gates,  1989).  The  natural  question  to  ask  at  this  point  is  whether  or  not 
the  recommendations  were  effective  at  reducing  any  of  the  perceived 
inefficiencies  of  the  DoD’s  acquisition  system.  Specifically,  were  cost  overruns 
reduced  as  a  result  of  recommendations  implemented  as  a  result  of  the 
Commission’s  study? 
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Cost  overruns  could  result  from  mismanagement  or  fraud;  although,  we 
noted  that  only  a  small  portion  of  cost  overruns  actually  seem  to  fall  in  this 
category.  This  view  would  be  consistent  with  the  thinking  that  cost  overruns  are 
a  direct  indication  of  program  inefficiencies.  Recall  that  the  appropriate  response 
in  this  case  would  be  to  improve  DoD’s  cost  management  capabilities  and  to 
improve  the  planning  process.  The  Commission’s  recommendations  seem  to 
imply  that  cost  overruns  are  actually  caused  by  mismanagement  or  fraud. 
Furthermore,  it  appears  that  the  intent  of  the  Commissions  recommendations 
was  to  nourish  a  new  acquisition  culture  within  the  Department  of  Defense  so 
that  decisions  on  purchasing  major  weapon  systems  were  based  on  realistic 
program  information.  If  this  is  actually  the  case  (that  overruns  are  due  to 
management  inefficiencies)  we  would  expect  cost  overruns  to  decrease  as  a 
result  of  these  reform  measures.  However,  it  is  important  to  understand  that 
these  measures  will  not  reduce  cost  overruns  if  they  are  caused  by  selection 
biases,  over-optimistic  projections,  or  some  other  factor. 

In  a  GAO  report  from  the  National  Security  and  International  Affairs 
Division  to  Senator  William  Roth  it  was  reported  that  President  Bush,  in  February 
1989,  directed  the  Secretary  of  Defense  to  develop  a  plan  to  fully  implement  the 
Commission’s  recommendations,  as  well  as  to  substantially  improve  defense 
management  overall  (GAO,  1990:1-2).  As  a  result,  in  July  1989,  the  President 
approved  and  the  Secretary  issued  the  Defense  Management  Report  (DMR) 
which  outlined  actions  needed  to  improve  DoD  management.  This  1990  report 
stated  that  the  DoD’s  efforts  to  implement  the  DMR  initiatives  were  in  various 
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stages.  It  stated  that  a  number  of  initiatives  were  already  undenway  and  that  the 
remaining  initiatives  were  all  planned  for  implementation  in  the  near  term.  A 
1991  GAO  report  by  the  same  office  stated  that  most  of  the  recommended 
changes  to  the  acquisition  system  had  been  made;  however,  it  also  pointed  out 
that  it  was  unclear  whether  the  initiatives  alone  would  bring  about  the  cultural 
transformation  intended  by  the  Packard  Commission  (GAO,  1991).  The  report 
noted  that  “the  success  of  these  changes  will  also  depend  on  strong  central 
leadership  from  the  Secretary  of  Defense,  strengthened  internal  controls  in  the 
acquisition  decision-making  process,  and  the  free  flow  of  information  both  up  and 
down  the  organization”  (GAO,  1991:1-2). 

The  problem  of  cost  overruns  is  multifaceted,  but  researchers  believe  that 
progress  can  be  made  by  attacking  its  various  aspects.  One  must  understand, 
though,  that  the  phenomenon  of  cost  overruns  (or  cost  growth  for  that  matter)  is 
not  likely  to  be  entirely  eliminated  by  partial  solutions  although  they  can  be  very 
helpful  in  reducing  the  proportions  of  cost  growth  exposure  (Scott,  1983:45). 
“Some  of  the  unpredictable  aspects  of  cost  growth  and  the  psychological  factors 
involved  are  uncontrollable  in  nature,  and  in  some  respects  are  linked  to  the 
inevitability  of  progress”  (Scott,  1983:45).  Nevertheless,  the  Department  of 
Defense  must  make  continuing  efforts  to  minimize  cost  overruns  on  major 
weapon  system  programs  by  exercising  proper  planning  and  management 
techniques  and  by  promoting  a  better  understanding  of  the  root  causes  of  cost 
overruns. 
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Summary 


The  phenomenon  of  cost  overruns  in  the  Department  of  Defense  has 
been  a  problem  for  decades.  In  fact,  regulations  to  control  defense 
procurements  extends  as  far  back  as  the  1940’s.  These  unplanned  cost 
increases  in  the  DoD’s  weapon  system  procurements  can  escalate  to  staggering 
amounts  and  can  impact  not  only  the  operational  users  but  the  people  of  this 
country  as  well.  Furthermore,  cost  growth  can  affect  our  national  security. 
Unfortunately,  cost  growth  has  persisted  despite  numerous  attempts  to  control 
and  mitigate  it.  Some  confusion  over  the  magnitude  of  the  problem  can  be 
traced  to  how  researchers  have  typically  measured  it.  Some  studies  have 
measured  cost  growth  while  others  have  measured  cost  overruns,  but  these 
measures  are  clearly  distinct.  Decision  makers  need  to  understand  this 
difference  when  determining  whether  a  cost  growth  of  20  percent  or  a  cost 
overrun  of  40  percent  is  significant  and  signaling  an  inefficiency  as  the  two 
measures  are  entirely  different.  Cost  growth,  unlike  the  cost  overrun  model, 
incorporates  cost  increases  due  to  risk  elements  and  change.  Not  only  are  these 
elements  unavoidable,  but  studies  have  shown  no  correlation  between  these 
changes  and  poor  cost  performance.  For  this  reason  this  study  will  use  the  cost 
overrun  model  as  it  represents  a  more  realistic  and  practical  measurement.  It’s 
practical  because  it  measures  cost  increases  resulting  from  controllable  factors. 

In  other  words,  it  allows  the  DoD  to  concentrate  on  problems  it  can  do 
something  about. 
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In  light  of  the  information  presented,  though,  one  should  consider  the 
nature  of  the  recommendations  made  by  the  Packard  Commission.  The 
recommendations  seem  to  focus  on  correcting  perceived  management 
inefficiencies  suggesting  that  cost  growth  results  from  mismanagement  or  fraud. 
These  recommendations  are  strikingly  similar  to  past  reforms  (Gates,  1989:7). 
Thus,  two  major  concerns  arise.  First,  we’ve  shown  that  little  progress  has  been 
made  over  the  years  in  reducing  the  problem  of  cost  growth.  Studies  have 
indicated  that  cost  growth  has  fluctuated  around  20  to  40  percent  since  the  mid 
1960’s  and  little  improvement  has  occurred  over  time.  If  the  Packard 
recommendations  are  similar  to  prior  reform  efforts  which  were  ineffective  should 
the  DoD  realistically  expect  improvements  in  cost  performance?  Secondly,  one 
should  be  concerned  about  the  focus  of  the  recommendations.  The 
recommendations  made  by  the  Packard  Commission  seem  to  focus  on 
management  inefficiencies  as  the  primary  factor  affecting  cost  growth.  However, 
studies  have  shown  that  the  minor  problems  of  fraud,  waste,  and  abuse,  or 
simply  poor  management,  actually  account  for  only  a  relatively  small  portion  of 
the  cost  growth  (Dreznerand  others,  1993:xi-xv;  Scott,  1983:37-38;  Weida, 

1 987:Ch  9).  If  management  inefficiencies  represent  such  a  small  portion  of  the 
problem  why  is  the  focus  of  the  Commission’s  reform  efforts  on  them?  Perhaps 
the  DoD’s  reform  efforts  would  be  more  effective  were  they  focused  more  on 
overcoming  the  various  planning  difficulties. 
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Propositions 


Although  the  Packard  Commission  released  their  recommendations  in 
1986  the  two  GAO  reports  from  the  National  Security  and  International  Affairs 
Division  suggest  that  the  recommendations  were  not  substantially  implemented 
until  1991  (GAO,  1990;  GAO,  1991).  This  thesis  proposes  to  test  the  following 
propositions  to  show  that  the  recommendations  implemented  as  a  result  of  the 
Packard  Commission  study  are  having  little  to  no  effect  on  reducing  cost 
overruns  on  major  weapon  system  contracts. 

•  Proposition:  Cost  overruns  for  completed  contracts  before  and  after 
1991  are  essentially  the  same.  The  Packard  recommendations  are 
similar  to  past  reform  efforts  which  have  been  ineffective,  and  they 
appear  to  target  those  factors  accounting  for  a  relatively  small  portion 
of  cost  overruns,  namely,  management  inefficiencies. 

•  Proposition:  The  recommendations  will  affect  contracts  differently 
based  on  contract  phase,  that  is,  whether  the  contract  is  for 
development  or  production.  While  significant  changes  (reductions)  are 
not  expected  to  occur  as  a  result  of  the  recommendations,  the  policy 
changes  are  expected  to  have  a  greater  impact  on  contracts  in  the 
development  phases  than  on  those  the  production  phases. 
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The  presence  of  cost  overruns  is  a  recurring  problem  in  the  DoD,  and  the 
study  of  possible  effects  resulting  from  the  implementation  of  acquisition  reform 
efforts  such  as  the  Packard  Commission  study  could  alter  this  trend.  In  the 
following  chapter  the  propositions  listed  above  will  be  more  clearly  delineated  in 
the  form  of  hypothesis  which  can  be  tested  using  the  presented  methodology. 
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III.  Methodology 


Overview 

This  chapter  describes  the  methodology  used  to  answer  the  investigative 
questions  fonwarded  in  Chapter  I.  Recall  the  basic  hypothesis  is  that  cost 
overruns  were  reduced  as  a  result  of  the  changes  implemented  from  the  Packard 
Commission’s  recommendations.  This  study  relies  on  the  comparison  of  means 
as  the  statistical  method  employed  to  determine  if  differences  exist  in  the 
population  means  before  and  after  the  changes  were  made.  As  such,  the 
methodology  structure  is  based  on  an  ex-post  facto  pseudo-experimental  design, 
or  simply  a  pre-test  -  post-test  experiment  with  no  control  group  relying  on 
archival  data.  A  similar  approach  was  used  effectively  in  a  1993  study 
examining  cost  overruns  between  two  populations  of  contracts  (Wandland  & 
Wickman,  1993). 

Data  Collection 

The  method  applied  in  this  study  relied  on  data  contained  in  the  Defense 
Acquisition  Executive  Summary  (DAES)  database.  This  database  is  maintained 
by  the  Office  of  the  Under  Secretary  for  Defense  for  Acquisition  &  Technology 
(OUSD(A&T))  and  contains  cost  performance  data  on  completed  contracts 
extending  back  as  far  as  1977  (DODI  5000.2,  1996:Part  6;  Christensen, 

1993:45;  Wilson,  1992:42).  Furthermore,  every  branch  of  the  service  reports 
summary  cost  data  to  the  database.  Because  of  this  broad  and  extensive 
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coverage  the  database  can  provide  program  managers  with  a  valuable 
management  tool  which  is  precisely  its  intent.  Specifically,  “the  purpose  of  the 
DAES  report  is  to  highlight  both  potential  and  actual  program  problems  to  the 
USD(A&T)  before  they  become  significant”  (DODI  5000.2,  1996:Part  6).  The 
DAES  database  available  for  this  study  contains  cost  performance  data  from 
1977  though  1995. 

Defense  contractors  regularly  report  cost  data  to  the  OUSD(A&T)  using  a 
number  of  cost  management  reports,  one  of  which  is  the  Cost  Performance 
Report  (CPR)  The  CPR  is  required  to  be  submitted  by  contractors  managing 
DOD  contracts  that  require  compliance  with  the  Earned  Value  Management 
Systems  Criteria.  EVMSC  establishes  uniform  evaluation  standards  for 
contractor’s  internal  management  control  systems.  Its  purpose  is  to  standardize 
performance  data  reporting,  to  provide  the  contractor  and  the  Government 
Program  Managers  with  accurate  and  reliable  data  to  monitor  execution  of  their 
program,  and  to  provide  a  basis  for  responsible  decision  making  (DODI  5000.2, 
1996; Part  3,9;  AFMCP  173-5,  1997).  Furthermore,  DODI  5000.2  states  that  this 
reporting  is  required  of  all  significant  contracts  and  subcontracts  within  all 
acquisition  programs.  “Significant  contracts  include  research,  development,  test, 
and  evaluation  contracts  and  subcontracts  with  a  value  of  $70  million  or  more  or 
procurement  contracts  and  subcontracts  with  a  value  of  $300  million  or  more  (in 
constant  FY’96  dollars)”  (DODI  5000.2,  1996:Part  3,10). 
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The  data  contained  in  the  CPR’s  are  summarized  and  sent  on  a  quarterly 
basis  to  the  OUSD(A&T)  (DODI  5000.2,  1996:Part  6).  This  regular  reporting 
ensures  early  indicators  of  contract  cost  and  schedule  problems  are  highlighted, 
and  it  displays  “the  effects  of  management  action  taken  to  resolve  problems 
affecting  cost  and  schedule  performance”  (DODI  5000.2,  1996:Part  6,11).  The 
intent  of  this  entire  cost  management  program  is  to  correct  problems  before  they 
become  serious.  Timely  analysis  is  the  key. 

The  CPR  contains  a  number  of  data  fields  four  of  which  form  the 
foundation  of  earned  value  measurement  and  cost  performance  reporting.  These 
four  principal  fields  are  listed  below  and  are  defined  according  to  the  DSMC 
Glossary  of  Defense  Acquisition  Acronyms  and  Terms  (DSMC,  1991).  Later  in 
this  chapter  equations  will  be  presented  demonstrating  how  these 
measurements  are  used  to  calculate  the  cost  variance  at  completion  ,or  the  Final 
Overrun  (FO),  and  the  Final  Overrun  Percentage  (FO%). 

Actual  Cost  of  Work  Performed  (ACWP):  The  cost  incurred  and 
recorded  in  accomplishing  the  work  performed  within  a  given  time  period. 
Budgeted  Cost  of  Work  Performed  (BCWP):  The  sum  of  the  budgets 
for  completed  work  packages  and  completed  portions  of  open  work 
packages,  plus  the  applicable  portion  of  the  budgets  for  level  of  effort  and 
apportioned  effort.  This  is  also  known  as  the  Earned  Value. 
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Budgeted  Cost  of  Work  Scheduled  (BCWS):  The  sum  of  budgets  for  all 
work  packages,  planning  packages,  etc.,  scheduled  to  be  accomplished 
(including  in-process  work  packages),  plus  the  amount  of  level  of  effort 
and  apportioned  effort  scheduled  to  be  accomplished  within  a  given  time 
period. 

Budget  at  Completion  (BAC):  The  sum  of  budgets  for  all  work  packages, 
planning  packages,  etc.,  for  the  entire  contract  excluding  the  management 
reserve  budget. 

Research  Design 

As  discussed  in  Chapter  II  changes  implemented  as  a  result  of  the 
Packard  Commission  recommendations  were  substantially  completed  by  1991. 
As  with  similar  policy  changes  as  broad  and  encompassing  as  these  there  is  no 
“on/off  switch;  changes  such  as  these  occur  over  time,  and  it  is  nearly 
impossible  to  determine  a  precise  date  of  implementation  for  the  aggregate 
change  (GAO,  1990;  GAO,  1991).  In  1990  Congress  directed  the  General 
Accounting  Office  (GAO)  to  study  and  report  on  the  DoD’s  progress  in 
implementing  the  Packard  Commission’s  recommendations  and  to  determine 
whether  the  intended  changes  had  occurred.  The  1990  report  stated  that  the 
DoD’s  efforts  to  implement  the  changes  outlined  in  the  1989  Defense 
Management  Report  were  in  various  stages,  that  a  number  of  initiatives  were 
already  underway,  and  that  the  remaining  initiatives  were  all  planed  for 
implementation  in  the  “near  term”  (GAO,  1990:1-2).  The  second  report  dated 
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August  1991  stated  that  most  of  the  recommended  changes  to  the  acquisition 
system  had  been  made  and  the  remaining  changes  would  be  completed  in  the 
short  term  (GAO,  1991).  For  the  purposes  of  this  study  the  Packard  Commission 
recommendations  are  considered  fully  implemented  as  of  December  31 , 1991 . 
This  date  is  then  set  as  the  “cut-off”  date  for  the  treatment  date  and  allows 
approximately  four  months  (Sept  -  Dec  1991)  for  the  remaining  few  changes  to 
have  been  completed.  The  reader  must  understand  that  this  date  was  selected 
judgmentally  based  on  the  available  literature.  Therefore,  this  study  will  compare 
the  population  of  contracts  completed  before  Dec.31,1991  with  the  population  of 
completed  contracts  following  that  date.  Specifically  this  thesis  will  use  various 
statistical  techniques  to  test  samples  from  the  two  populations  (before  and  after) 
in  order  to  make  inferences  about  the  difference  between  population  parameters 
if  one  exists. 

Research  Population 

Using  Dec  31, 1991  as  our  treatment  date  allows  for  approximately  four 
years  of  post-treatment  data  to  be  included  in  this  study.  This  study  uses  the 
DAES  database  to  capture  cost  information  on  contracts  completed  in  the  period 
of  January  1,  1988  to  December  31,  1995.  Any  contracts  not  completed  in  this 
period  were  then  eliminated  from  the  population.  This  eight  year  period  provides 
for  approximately  four  years  of  pre  and  post  treatment  data,  and  the  length  of 
time  should  mitigate  any  potential  bias  due  to  factors  such  as  fluctuations  in  the 
defense  business  cycle  (Wandland  &  Wickman,  1993:28-29).  A  1993  RAND 
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study  on  cost  growth  briefly  addressed  the  question  of  whether  or  not  cost 
growth  has  improved  overtime.  However,  that  study  did  not  address  the 
possible  effects  of  the  Packard  recommendations,  it  had  at  most  one  to  two 
years  of  data  available,  and  it  measured  cost  growth  as  opposed  to  cost 
overruns  (Drezner  and  others,  1993).  Furthermore,  the  time  trend  study  used 
data  only  through  1989.  As  the  Packard  recommendations  were  not  fully 
implemented  until  approximately  December  1991  any  possible  effects  would  not 
be  evident  in  that  particular  study.  This  study  overcomes  these  shortfalls,  and 
the  results  should  significantly  benefit  policy  makers.  Also,  only  those  contracts 
in  the  DAES  database  that  reported  a  BAG  and  ACWP  were  retained  as  these 
cost  figures  are  required  to  determine  the  final  overrun.  The  contracts  were  then 
classified  by  program  phase  in  order  to  answer  the  research  questions  and  to 
test  the  hypotheses.  Program  phase  was  broken  down  into  two  phases: 
development  and  production.  These  two  phases  capture  the  “totality  of  system 
acquisition  programs  under  this  category.  In  DoD  programs  systems  are  either 
development  or  production  with  little  or  no  discrepancy  between  them"  (Wilson, 
1992:48).  As  systems  progress  from  the  development  to  production  phases  the 
performance,  schedule,  and  cost  parameters  become  more  predictable  and  less 
variable  as  the  risks  and  uncertainties  decrease  over  time.  As  a  result  the  cost 
overruns  should  be  distributed  differently  in  the  two  groups.  One  would  expect 
then  that  the  policy  changes  would  have  a  greater  impact  on  contracts  in  the 
development  phases  than  those  in  production  phases.  Any  contract  not 
specifying  program  phase  was  eliminated  from  the  population  as  well.  Less  than 
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one  percent  of  the  contracts  completed  in  the  eight  year  period  of  interest  failed 
to  report  BAG  or  ACWP  figures  or  specify  program  phase;  furthermore,  these 
few  omissions  were  evenly  distributed  across  development  and  production 
contracts  and  between  the  before  and  after  groups.  As  such,  selection  bias  is 
minimal  and  the  results  should  be  generalizable  and  externally  valid. 

Completed  Contract  Defined.  This  study  limited  the  population  of 
contracts  examined  to  completed  contracts.  For  the  purposes  of  this  study  a 
completed  contract  is  one  that  is  at  least  75  percent  complete.  An  empirical 
examination  of  the  DAES  database  suggests  that  many  contractors  discontinue 
CPR  submissions  when  the  contract  is  at  or  near  completion.  At  approximately 
the  75  percent  completion  point  and  beyond  contract  cost  performance  is 
relatively  stable  (Gordon,  1996:Ch  3-4;  Wilson,  1992:Ch  3).  The  majority  of 
costs  are  incurred  when  a  contract  is  from  15  to  75  percent  complete.  This  is 
readily  visible  upon  examination  of  the  “S-curve”  that  is  generated  when  the 
percent  of  costs  incurred  is  plotted  against  the  corresponding  percent  of  contract 
completion  (Weida,  1977).  The  “S”  shaped  curve  represents  the  “normal  ramp  up 
and  ramp  down  of  a  typical  program”  (Wilson,  1992:37).  Figure  3  demonstrates 
this  concept.  Notice  how  the  cost  curve  flattens  out  as  the  program  nears 
completion.  It  may  be  for  this  reason  that  many  contractors  tend  to  discontinue 
submitting  CPRs  around  the  75%  completion  point.  Practically  speaking  the 
contract  is  nearly  complete  at  this  point  in  terms  of  the  costs  incurred 
(Christensen,  1997).  In  essence,  by  limiting  this  study  to  contracts  that  are  75% 
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complete  the  results  of  this  thesis  will  be  relatively  conservative  as  cost 
performance  beyond  the  75%  point  has  been  shown  to  be  fairly  stable.  Even  if 
performance  doesn’t  worsen  it  will,  at  most,  remain  unchanged. 


Typical  "S"  Shaped  Cost  Curve 
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Figure  3.  “S”  Curve 


Sample  Defined.  The  contracts  examined  in  this  study  represent  only  a 
portion  of  those  contracts  contained  in  the  DAES  database.  First  of  all,  while 
the  DAES  database  does  contain  cost  performance  data  reported  on  sensitive 
and  highly  classified  contracts  that  type  of  data  is  not  included  in  this  study. 
Sensitive  and  highly  classified  contracts  are  managed  under  separate  cost 
reporting  requirements  and  are,  therefore,  not  considered  to  be  a  part  of  the 
population  of  contracts  examined  in  this  study.  Secondly,  any  contracts  that  may 
have  been  canceled  prior  to  the  75  percent  completion  point  would  be  excluded 
from  the  sample.  Furthermore,  one  should  understand  that  the  definition  relied 
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upon  for  a  “completed  contract”  reduces  the  usable  database.  Only  269  out  of 
the  1,843  individual  contracts  contained  in  the  database  were  retained  for 
examination.  Finally,  any  contracts  not  reporting  BAG  and  ACWP  figures  or 
specifying  program  phase  have  been  eliminated  from  the  population  as  well. 
Thus,  although  some  selection  bias  may  be  present  in  the  study  it  is  minimal.  As 
such,  the  sample  used  in  this  research  should  represent  the  entire  population  of 
significant  DOD  contracts  and  subcontracts  within  all  acquisition  programs  as 
defined  in  DODI  5000.2.  One  can  assume  then  that  any  effects  identified  in  this 
sample  would  be  representative  of  the  entire  population. 

Table  4  describes  the  population  of  contracts  examined  in  this  study.  The 
table  displays  the  number  of  contracts  contributed  by  all  services  to  the  overall 
population  in  each  of  the  categories;  before  and  after  Packard  and  development 
or  production.  Also  shown  by  category  are  the  average  ACWP’s,  BAG’S,  and 
Final  Overrun’s  and  their  associated  standard  errors.  A  complete  listing  of 
contracts  is  shown  in  the  appendix. 

Table  4.  Population  Of  Contracts  Described 


Before  Packard 

development  production  subtotal 

After  Packard 

development  production  subtotal 

Total 

number 

47 

101 

148 

34 

87 

121 

269 

average  ACWP 

311.57 

377.11 

356.30 

768.38 

269.00 

576.68 

452.90 

standard  error 

597.94 

1276.40 

1105.04 

2010.85 

426.10 

1121.44 

1115.47 

average  BAG 

294.30 

363.10 

341.25 

647.03 

458.14 

515.91 

417.70 

542.21 

1231.90 

1060.85 

1661.21 

389.53 

935.46 

1008.07 

-17.28 

-15.05 

-121.35 

-35.80 

-60.78 

-35.20 

standard  error 

59.35 

53.43 

55.20 

499.44 

84.22 

274.17 

189.28 

-averages  and  standard  errors  reported  in  millions 
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Research  Variable  Defined 


This  study  examines  cost  overruns  on  completed  contracts,  or  the  “final 
overrun,"  as  opposed  to  the  “current  overrun”  which  is  the  adverse  cost  variance 
to  date  on  a  contract  in  progress.  The  final  cost  overrun  is  then  measured  as  the 
difference  between  the  budget  for  all  the  work  on  the  contract,  the  BAG,  and  the 
actual  cost  of  the  work  performed  (ACWP)  as  of  the  time  the  CPR  was 
submitted.  If  a  variance,  or  a  difference,  exists  between  the  ACWP  and  the 
BAG  then  the  contractor  either  spent  more  or  less  money  than  what  was 
planned.  Either  case  is  of  concern  if  the  variance  is  significant. 

As  defined  in  Chapter  II  the  overrun  at  completion  is  calculated  as  the 
difference  between  the  BAG  and  the  EAC.  This  is  the  same  formula  as 
described  above  as  the  BCWP  of  a  completed  contract  is  the  same  as  the  BAG, 
and  the  ACWP  of  a  completed  contract  equals,  or  rather  replaces,  the  EAC.  The 
ACWP  reflects  the  total  costs  incurred  and  is  not  simply  an  estimate.  For  the 
purposes  of  this  study,  the  final  overrun,  or  cost  variance,  can  then  be  calculated 
as  shown  in  Equation  3.  Thus,  a  negative  final  overrun  figure  will  indicate  the 
occurrence  of  an  overrun;  positive,  an  underrun. 


Final  _  Final 
Overrun  BAC 


-ACWP 


(3) 


This  final  overrun  (FV)  measure  can  be  misleading.  Final  overrun, 
calculated  as  shown  above,  fails  to  consider  contract  size  (in  terms  of  total  dollar 
value)  and  inflation.  Certainly  one  cannot  reasonably  compare  a  $1  million  dollar 
overrun  on  a  $100  million  dollar  contract  with  a  $1  million  dollar  overrun  on  a  $2 
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million  dollar  contract.  In  both  cases  the  overrun  is  $1  million;  ho\wever,  that 
figure  represents  a  one  percent  overrun  in  the  first  scenario  compared  to  a  50 
percent  overrun  in  the  second.  Also,  the  effects  of  inflation  over  the  years  must 
be  accounted  for.  By  converting  the  final  overrun  figure  to  a  final  overrun 
percentage  one  can  adjust  the  figure  for  size  and  inflation  thus  providing  a 
relative  measure  of  cost  performance.  The  final  overrun  percentage  (FO%)  can 
be  calculated  as  shown  in  Equation  4. 


OverrufiFinat 

BACf.. 


(4) 


The  methodology  used  in  this  thesis  uses  the  mean  FO%  of  each  population  as 
the  parameter  of  study.  For  the  purposes  of  this  study  the  mean  FO%  is 
calculated  is  shown  in  Equation  5. 


Mean  FO%  =  ^ - 

«  (5) 

where  /  =  the  rth  contract  in  the  population  and  n  =  the  total  number 
of  observations  (contracts)  in  the  population. 


Statement  of  Hypotheses 

The  research  questions  and  the  hypotheses  tested  are  listed  below. 


Research  Question  #1:  Is  the  mean  final  overrun  percentage  (FO%)  for 
contracts  completed  before  Dec.31,1991  different  than  the  mean  variance 
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percentage  afterwards?  If  the  following  hypothesis  is  true  we  can  assume 
that  the  Packard  recommendations  were  ineffective  at  reducing  cost 
overruns.  If  the  following  hypothesis  is  rejected  we  can  assume  that  cost 
performance  changed  (for  the  better  or  for  the  worse)  as  a  result  of  the 
recommendations. 

Hypothesis  1: 

Hj).  mean  FO%  (FO%p,)  before  Dec  3I'  1991  “  FO /bp  afterwards 
Ha-  FO%p  before  Dec  31, 1991  ^  FO /bp  afterwards 

Research  Question  #2:  Does  the  mean  FO%  for  contracts  completed 
before  and  after  Dec.31,1991  differ  between  program  phase?  If  the 
following  hypothesis  is  true  we  can  assume  that  the  Packard 
recommendations  were  ineffective  at  reducing  cost  overruns  in 
development  or  production  contracts.  If  the  following  hypothesis  is 
rejected  we  can  assume  that  cost  performance  changed  (for  the  better  or 
for  the  worse)  as  a  result  of  the  recommendations. 


Hypothesis  2: 

Hq-  ^^*^Fdevelopment/before  “  '^Fdevelopment/aftef 

Ha"  FO  /bpbjveiopmftnt/befofe  ^  FO  /bp(jgyeiopnTent/aftei. 


Hypothesis  3: 

Hq-  FO  /b Ppcobuction/before  “  FO  ^Pp^odllct^on/after 

Fig.  FO  /bpproductlon/befdre  ^  FO  /bpprobuctkxi/after 
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Method  of  Analysis 


Each  of  the  hypotheses,  generated  from  the  two  research  questions  will 
be  tested  by  statistical  analysis  using  one  or  both  of  the  models  identified  below. 
Research  question  #1  was  analyzed  in  both  phases.  Research  question  #  2  was 
analyzed  only  in  phase  two.  These  are  non-directional  tests.  Inferences  about 
direction  and  magnitude  of  any  effect  will  be  made  by  descriptive  analysis  of  the 
data. 

Phase  I  Model 

The  first  phase,  shown  in  Table  5,  tested  for  differences  in  the  mean  FO% 
between  contracts  completed  before  and  after  the  treatment  date  (December  31, 
1991). 


Table  5.  Phase  I  Model 


Before  Packard 

After  Packard 

n 

FO  /o  before  - 

n 

where  /  =  the  /th  contract  in  the 
before  treatment  population 
and  n  =  the  total  number  of 
observations  (contracts)  in  that 
population. 

FOVo  after  =  - 

n 

where  j  =  the  yth  contract  in  the 
after  treatment  population  and  n  = 
the  total  number  of  observations 
(contracts)  in  that  population. 
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Phase  2  Model 


The  second  phase  tested  for  differences  in  the  mean  FO%  between  program 
phase  (development  or  production)  contracts  completed  before  and  after  the 
treatment.  As  seen  in  Table  6  this  phase  is  essentially  testing  vertically  and 
included  two  tests. 

Test  1 :  Development,  Before  vs  After  Packard 
Test  2:  Production,  Before  vs  After  Packard 


Table  6.  Phase  II  Model 


Development 

Production 

Xfo%. 

FO/oprod  -  ' 

Before  Packard 

where  k  =  the  kth 

where  /  =  the  /th  production 

development  contract  in 

contract  in  the  before 

the  before  treatment 

treatment  population  and  n  = 

population  and  n  =  the 

the  total  number  of 

total  number  of 

observations  (contracts)  in 

observations  (contracts) 
in  that  population. 

that  population. 

n 

^FO%p 

dev  after  “  ^ 

FO%  prod  =  ^ 

afer 

After  Packard 

where  m  -  the  mth 

where  p  =  the  pth  production 

development  contract  in 

contract  in  the  after 

the  after  treatment 

treatment  population  and  n  = 

population  and  n  =  the 

the  total  number  of 

total  number  of 

observations  (contracts)  in 

observations  (contracts) 
in  that  population. 

that  population. 
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Tests  For  Difference  Between  Population  Means 


The  principal  statistical  analysis  used  in  this  study  tests  for  a  difference  in 
the  population  means.  Using  the  sample  mean  as  an  estimator  of  the  population 
mean  allows  for  inferences  to  be  made  about  the  differences  between  population 
parameters.  Because  the  sample  sizes  are  large  more  reliable  inferences  can 
be  made  with  fewer  assumptions  about  the  sampled  population  (McClave  & 
Benson,  1994:446).  The  t-test  is  a  parametric  test  which  can  be  used  to  detect  a 
difference  between  the  means  of  two  populations--if  such  a  difference  exists 
(McClave  &  Benson,  1994:Ch  9).  However,  in  order  to  use  the  t-statistic  a 
number  of  assumptions  must  be  satisfied:  both  sampled  populations  must  be 
approximately  normally  distributed  with  equal  population  variances,  and  the 
random  samples  must  be  selected  independently  of  each  other.  In  order  to 
analyze  data  from  populations  that  do  not  satisfy  these  assumptions  a 
nonparametric  statistical  method  such  as  the  Mann-Whitney  test  or  the  Kruskal- 
Wallis  H-test  must  be  used  (Connover,  1980:229).  These  two  tests  are  the 
nonparametric  counterparts  to  the  two  sample  t-test. 

The  Mann-Whitney  test  was  designed  to  test  two  independent  samples. 
The  Kruskal-Wallis  test  is  an  extension  of  the  Mann-Whitney  test  designed  to 
tackle  the  problem  of  analyzing  k  independent  samples,  for  k  >2  (Connover, 
1992:229,  McClave  &  Benson,  1994:Ch  17).  The  Kruskal-Wallis  test  uses  the 
same  information  contained  in  the  observations  as  does  the  Mann-Whitney  test 
and  studies  have  shown  that  for  two  samples  the  Kruskal-Wallis  test  is 
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equivalent  to  the  Mann-Whitney  test  (Connover,  1980:236,  McClave  &  Benson, 
1994:928). 

The  Mann-Whitney  test  “has  been  found  to  perform  particularly  well  as  a 
test  for  equal  means,  since  it  is  especially  sensitive  to  differences  in  location" 
(Gibbons,  1971:149).  In  fact,  “when  the  populations  are  assumed  to  differ  only 
in  location,  the  Mann-Whitney  test  is  directly  comparable  with  Student’s  t-test  for 
means”  (Gibbons,  1971:149).  Consequently,  many  statisticians  consider  the 
Mann-Whitney  test  the  best  nonparametric  test  for  location  (Gibbons,  1971:149). 
For  these  reasons  this  study  will  use  the  Mann-Whitney  test  as  the 
nonparametric  model  of  choice. 

The  Mann-Whitney  test  for  a  completely  randomized  design  is  a 
nonparametric  model  which  can  be  used  to  compare  two  populations  when  the  t- 
test  is  inappropriate  for  making  the  comparison  (Connover,  1980:229,  McClave  & 
Benson,  1994:928).  It  tests  the  null  hypothesis  that  the  two  populations  are 
identical  against  the  alternative  that  one  of  the  populations  tends  to  furnish 
greater  observed  values  than  the  other.  When  the  assumptions  are  appropriate, 
parametric  tests  are  generally  more  powerful  than  their  nonparametric 
equivalents,  although  nonparametric  tests  often  compare  quite  well  in 
performance  (McClave  &  Benson,  1994,  Ch  17;  Connover,  1980:1-4).  The 
parametric  models  test  hypotheses  regarding  the  group  means.  The 
nonparametric  methods,  on  the  other  hand,  test  central  value  hypotheses  based 
on  measures  other  than  the  mean  such,  for  example,  the  median  (McClave  & 
Benson,  1994:Ch  9  &  Ch  17).  When  the  assumptions  are  satisfied  this  study 
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uses  the  more  robust  parametric  method.  When  the  assumptions  are  not 
satisfied  the  nonparametric  method  is  used. 

Data  Preparation 

As  stated  previously,  this  study  limited  the  population  of  contracts 
examined  to  only  completed  contracts  -  all  others  were  discarded  from  the  study. 
These  completed  contracts  were  then  sorted  into  either  pre  or  post  treatment 
based  on  the  date  of  the  last  CPR  submitted.  They  were  then  further 
categorized  by  program  phase.  A  total  of  three  tests  were  conducted  in  order  to 
answer  the  research  questions:  one  in  phase  I  and  two  in  phase  2.  Tables  7  and 
8  display  the  assumptions  and  hypotheses  for  both  the  parametric  t-test  and  for 
the  nonparametric  Mann-Whitney  test. 


Table  7.  Assumptions 


1  t-test 

Mann-Whitney  test 

1 .  All  samples  are  random  samples 
from  their  respective  populations. 

1 .  All  samples  are  random  samples 
from  their  respective  populations. 

2.  All  samples  are  independently 
selected  from  their  respective 
populations. 

2.  All  samples  are  independently 
selected  from  their  respective 
populations. 

3.  Both  sampled  populations  have 
relative  frequency  distributions  that  are 
approximately  normal. 

3.  The  2  probability  distributions  from 
which  the  samples  are  drawn  are 
continuous. 

4.  The  population  variances  are  equal. 
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Table  8.  Hypotheses 


I  t-test 

Mann-Whitney  test 

Ho 

(Pi  -P2)  =  0 

The  2  sampled  populations  have 
identical  probability  distributions. 

Ha 

(Fi  -1^2)  *■  0 

The  probability  distribution  for 
population  A  is  shifted  to  the  left  or 
to  the  right  of  that  for  B 

Randomness.  The  assumption  of  a  random  sample  applies  to  both 
models.  “A  sample  from  a  finite  population  is  a  random  sample  if  each  of  the 
possible  samples  was  equally  likely  to  be  obtained”  (Connover,  1980:62).  As 
the  populations  being  compared  throughout  this  study  are  of  manageable  size 
the  entire  data  set  is  used  in  every  test.  The  use  of  the  entire  population, 
therefore,  satisfies  this  assumption.  Since  samples  don’t  have  to  be  obtained 
the  means  by  which  the  samples  are  obtained  (randomness)  is  not  a  factor. 

Independence.  This  assumption,  like  randomness,  applies  to  both 
models.  One  can  assume  that  the  population  of  major  weapons  systems 
contracts  reported  under  DODI  5000.2  contains  contracts  independent  of  each 
other.  First  of  all,  although  all  of  these  contracts  are  regulated  by  the  same 
legislation,  for  example,  the  Federal  Acquisition  Regulation,  or  the  FAR,  and 
instructions  such  as  DODI  5000.2,  all  of  the  contracts  are  DoD  contracts.  Any 
likeness  caused  by  particular  legislation  or  regulations  would  not  violate  the 
assumption  of  independence  as  these  laws  would  affect  cost  performance 
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equally  across  all  DoD  contracts.  Secondly,  the  database  used  in  this  study 
contains  cost  information  from  378  programs  encompassing  1 ,843  individual 
contracts.  These  programs  could  possibly  influence  the  cost  performance  of  the 
multiple  contracts  managed  under  their  umbrellas.  However,  although  a  large 
number  of  the  contracts  may  be  managed  by  the  same  program  personnel  and 
while  it  is  not  uncommon  for  a  particular  program  to  contain  dozens  of  individual 
contracts,  most  cost  management  activities  are  performed  by  a  multitude  of 
individual  contractors  not  by  one  particular  program  office.  For  these  reasons 
the  assumption  of  independence  can  be  established.  Readers  should  be  aware 
that  a  violation  of  this  assumption  leads  to  a  decrease  in  the  confidence  of  any 
inferences  made  from  the  test  results.  Due  to  the  exploratory  nature  of  this 
research,  a  level  of  significance  (a)  of  10  percent  is  established.  This  means 
that  the  probability  of  making  a  type  1  error  (rejecting  the  null  hypothesis  when  it 
is  in  fact  true)  is  10  percent. 

Model  Selection.  When  the  assumptions  are  appropriate,  parametric 
tests  are  generally  more  powerful  than  their  nonparametric  equivalents. 

(McClave  &  Benson,  1994:Ch  17)  Although  nonparametric  tests  often  compare 
quite  well  in  performance  the  preference  in  this  study  is  the  use  of  the  more 
powerful  t-test.  Therefore,  the  first  test  conducted  on  each  population  is  one  of 
normality.  A  number  of  tests,  both  graphical  and  non-graphical,  are  available  for 
assessing  normality  (Stevens,  1992:251).  One  of  the  popular  graphical 
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procedures  is  to  simply  examine  the  histogram  for  each  dependent  variable  in 
each  group.  “This  does  give  some  indication  whether  normality  might  be 
violated.  However,  with  small  or  moderate  sample  size,  it  is  difficult  to  tell 
whether  the  non-normality  is  real  or  apparent,  because  of  considerable  sampling 
error”  (Stevens,  1992:253).  For  this  reason  the  preference  among  researchers  is 
the  use  of  a  non-graphical  test. 

The  Shapiro-Wilk  test  for  normality  comes  highly  recommended  over  less 
rigorous  tests  such  as  the  chi-square  test  for  normality  or  the  Kolmogorov- 
Smirnov  test,  and,  overall,  it  is  probably  the  most  powerful  non-graphical  test  in 
detecting  departures  from  normality  (D’Agostino  &  Stephens,  1986:406;  Stevens, 
1 992:253).  For  these  reasons,  the  Shapiro-Wilk  test  is  used  to  test  for  normality 
in  the  population  distributions.  The  test  produces  an  approximate  Shapiro-Wilk 
test  statistic  ranging  from  0  to  1 .  A  small  value  for  the  statistic  (as  determined  by 
a  and  the  sample  size)  indicates  non-normality  (Connover,  1980:363-366).  The 
only  assumption  required  for  the  Shapiro-Wilk  test  is  that  the  sample  is  a  random 
sample.  The  level  of  significance  for  the  normality  tests  is  also  set  at  10  percent. 

If  the  population  has  a  normal  distribution  the  next  step  is  to  test  for  equal 
variance.  If  the  population  does  not  have  a  normal  distribution  the  test  for  equal 
variance  is  not  required,  and  the  Mann-Whitney  test  is  used,  (see  the  decision 
tree  in  Figure  3.2)  “The  common  statistical  procedure  for  comparing  population 
variances  (a^^  and  Q2)  makes  an  inference  about  the  ratio,  (oi^aj^),  based  on 
the  ratio  of  sample  variances,  s^^/Sj^”  (McClave  &  Benson,  1994:  412).  This  ratio 
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has  an  approximate  F  distribution.  When  the  population  variances  are  unequal 
the  ratio  F  of  the  sample  variances  is  expected  to  be  either  very  large  or  very 
small  (Connover,  1980:246-247;  McClave  &  Benson,  1994:412).  Two 
assumptions  are  required  for  the  F  test.  The  two  sampled  populations  must  be 
normally  distributed,  and  the  samples  are  randomly  and  independently  selected 
from  their  respective  populations.  The  level  of  significance,  a,  for  these  tests  is 
also  set  at  10  percent .  When  these  assumptions  are  satisfied  and  the  two 
population  variances  are  equivalent  the  sampling  distribution  of  F=  Si^/Sj^  is  the  F 
distribution  (McClave  &  Benson,  1994:412).  These  F  distributions  are  tabulated 
and  are  then  compared  to  the  calculated  F  statistics.  If  the  calculated  F  statistic 
is  greater  than  the  tabulated  value  then  the  null  hypothesis  of  equal  variances  is 
rejected.  If  the  populations  to  be  compared  have  normal  distributions  and  equal 
variances  then  the  t-test  is  conducted.  If  the  assumptions  of  normality  and  equal 
variance  are  not  satisfied  the  Mann-Whitney  test  is  used.  The  decision  tree 
shown  in  Figure  4  displays  the  logic  of  the  model  selection  used  in  this  thesis. 
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Figure  4.  Decision  Tree  For  Model  Selection 


Robustness 

The  tests  identified  above  are  set  up  as  mathematical  models  which  are 
based  on  the  assumptions  already  discussed.  As  "all  mathematical  models  are 
approximations  to  reality,  violations  of  the  assumptions  are  inevitable"  (Stevens, 
1992:237).  The  question  to  ask,  then,  is  "how  much  can  a  given  assumption  be 
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violated  before  it  affects  the  error  rate  (a)?".  The  t  and  F  tests  are  fairly  robust 
with  respect  to  the  normality  assumption,  but  only  when  the  samples  are  large 
(Stevens,  1992:238).  Robust  means  that  the  actual  error  rate  (a)  is  very  close  to 
the  nominal  a  (the  level  set  by  the  experimenter).  The  F  statistic  is  also  robust 
against  heterogeneous  variances  when  group  sizes  are  equal  (Stevens, 
1992:238-239). 

The  most  important  assumption,  however,  is  that  of  independence.  "It  is 
by  far  the  most  important  assumption  for  even  a  small  violation  of  it  produces  a 
substantial  effect  on  the  level  of  significance"  (Stevens,  1992:239).  Recall  that 
the  assumption  of  independence  is  required  for  both  the  parametric  t-test  and 
the  non-parametric  Mann-Whitney  model.  The  requirements  for  normality  and 
homogeneous  variance  in  the  parametric  model  will  be  held  to  strict 
interpretations  of  the  test  results  even  though  some  researchers  view  them  as 
fairly  robust  models  with  respect  to  normality  and  variance  (Stevens,  1992:239). 
The  rationale  used  here  is  that  any  violations  simply  lead  to  a  further  decrease  in 
the  confidence  of  inferences  made  from  the  test  results.  The  decision  tree 
displayed  in  Figure  4  displays  the  model  selection  process  that  will  be  followed  in 
this  thesis. 
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Summary 


The  DAES  database  provides  the  researcher  with  summary  cost 
performance  data  on  significant  defense  contracts  and  allows  for  the  ready 
calculation  of  cost  overruns  or  cost  variances.  The  various  data  fields  presented 
in  the  database  also  allow  the  researcher  to  categorize  these  contracts  by 
factors  such  as  completion  date,  program  phase,  and  branch  of  service  among 
others.  This  categorization  allows  the  researcher  to  more  thoroughly  examine  the 
possible  effects  of  treatments,  such  as  policy  changes,  on  cost  performance,  and 
it  supports  the  hypothesis  testing  required  to  answer  the  research  questions  of 
this  thesis.  Using  parametric  and  nonparametric  statistical  methods  we  can 
determine  whether  or  not  differences  exist  between  the  mean  final  overrun 
percentages  of  the  before  and  after  groups.  If  the  changes  implemented  as  a 
result  of  the  Packard  Commission’s  recommendations  were  effective  in  reducing 
cost  overruns  a  difference  in  the  mean  final  overrun  percentage  should  be  noted 
between  the  population  of  contracts  completed  before  and  after  the  change. 
Specifically,  the  mean  final  overrun  percentage  should  increase  as  a  result  of  the 
treatment. 
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IV.  Results 


Statistical  Results  of  the  Hypothesis  Tests 

The  chapter  presents  the  results  of  the  statistical  tests  conducted  to  answer 
the  two  research  questions  presented  in  Chapters  I  and  III.  The  results  are 
presented  by  phase,  and  an  analytical  summary  for  all  tests  are  presented  in  Table  9 
at  the  end  of  this  chapter. 

Data  Description  of  the  Phase  I  Model.  The  first  phase  tested  for 
differences  in  the  mean  FO%  between  contracts  completed  before  and  after  the 
treatment  date  (December  31.  1991).  The  null  hypothesis  states  that  the  mean  FO% 
before  the  treatment  date  is  equal  to  the  mean  FO%  afterwards  while  the  alternate 
hypothesis  states  that  the  FO%  before  is  different  than  that  aften/vards.  Table  9  on 
page  74  displays  the  two  populations  of  contracts  examined  in  the  first  test.  This  test 
compared  the  population  of  contracts  completed  before  the  Packard 
recommendations  with  the  population  of  contracts  completed  afterwards.  Figures  5 
and  6  show  a  graphical  representation  of  the  two  populations  in  the  form  of  two 
histograms.  A  smooth  line  representing  a  normal  probability  distribution  is  overlaid 
on  both  distributions  allowing  the  reader  to  graphically  compare  the  actual 
distributions  with  approximately  normal  distributions. 
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Test  of  Normality  Results.  The  null  hypothesis  for  the  Shapiro-Wilk  test 
states  that  the  distribution  is  normal  while  the  alternate  hypothesis  states  that  it  is 
non-normal.  The  only  assumption  required  for  the  test  is  that  the  sample  is  a 
random  sample,  and  the  level  of  significance  (a)  for  the  normality  test  was  set  at  10 
percent.  The  Shapiro-Wilk  test  performed  on  the  population  of  contracts  completed 
before  Packard  produced  a  test  statistic  of  .8464.  The  p-value  associated  with  this 
test  statistic  was  approximately  zero  indicating  non-normality  in  the  distribution.  The 
Shapiro-Wilk  test  performed  on  the  population  of  contracts  completed  after  Packard 
produced  a  test  statistic  of  .833.  The  p-value  associated  with  this  test  statistic  was 
also  approximately  zero  indicating  that  this  distribution  was  non-normal  as  well. 

Both  of  the  populations  to  be  compared  appear  to  have  non-normal 
distributions  at  the  90  percent  level  of  confidence.  This  is  evident  graphically  as  well. 
The  problem  of  skewness  and  kurtosis  in  the  two  distributions,  as  shown  in  Figures  5 
and  6,  provides  graphical  support  of  the  non-normality  results  obtained  from  the 
Shapiro-Wilk  test.  As  such,  the  test  for  equal  variance  was  not  required,  and  the 
non-parametric  Mann-Whitney  test  was  selected  to  test  the  hypothesis. 

Results.  The  Mann-Whitney  model  tests  the  null  hypothesis  which  states  that 
the  two  sampled  populations  have  identical  probability  distributions  against  the 
alternate  hypothesis  which  states  that  the  probability  distribution  for  population  A  (the 
population  before  Packard)  is  shifted  to  the  right  of  that  for  population  B  (the 
population  after  Packard).  The  test  performed  on  the  two  distributions  produced  a 
test  statistic  of  7941.0  which  has  an  associated  p-value  of  .0553347.  As  the  level  of 
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significance  for  this  test  was  established  at  10  percent  (a=.10)  the  results  suggests 
that  sufficient  evidence  exists  to  reject  the  null  hypothesis  at  the  90  percent  level  of 
confidence.  The  two  distributions  may  actually  differ  in  location.  In  other  words,  the 
mean  percentage  overrun  for  contracts  completed  after  the  treatment  appears  to  be 
statistically  different  than  the  mean  percentage  overrun  for  the  earlier  group.  In  fact, 
the  distribution  of  the  population  of  development  contracts  completed  after  Packard 
seems  to  be  shifted  to  the  left  of  the  earlier  population.  Othenwise  stated,  the  mean 
percentage  overruns  seems  to  have  worsened  after  the  policy  went  into  effect. 
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Data  Description  of  the  Phase  2  Model.  The  second  phase  tested  for 
differences  in  the  mean  FO%  between  program  phase  (development  or  production) 
contracts  completed  before  and  after  the  treatment  date  (December  31, 1991).and 
included  two  tests.  The  null  hypothesis  for  the  first  test  states  that  the  mean  FO% 
of  development  contracts  completed  before  the  treatment  date  is  equal  to  the  mean 
FO%  of  development  contracts  completed  afterwards  while  the  alternate  hypothesis 
states  that  the  FO%  before  is  different  than  that  afterwards.  Table  9  displays  the  two 
populations  of  contracts  examined  in  the  first  test.  This  test  compared  the  population 
of  development  contracts  completed  before  the  Packard  recommendations  with  the 
population  of  development  contracts  completed  afterwards.  Figures  7  and  8  show  a 
graphical  representation  of  the  two  populations  in  the  form  of  two  histograms.  A 
smooth  line  representing  a  normal  probability  distribution  is  overlaid  on  both 
distributions  allowing  the  reader  to  graphically  compare  the  actual  distributions  with 
approximately  normal  distributions. 

Test  of  Normality  Results.  The  null  hypothesis  for  the  Shapiro-Wilk  test 
states  that  the  distribution  is  normal  while  the  alternate  hypothesis  states  that  it  is 
non-normal.  The  only  assumption  required  for  the  test  is  that  the  sample  is  a 
random  sample,  and  the  level  of  significance  (a)  for  the  normality  test  was  set  at  10 
percent.  The  Shapiro-Wilk  test  performed  on  the  population  of  development 
contracts  completed  before  Packard  produced  a  test  statistic  of  .8073.  The  p-value 
associated  with  this  test  statistic  was  approximately  zero  indicating  non-normality  in 
the  distribution.  The  Shapiro-Wlk  test  performed  on  the  population  of  contracts 
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completed  after  Packard  produced  a  test  statistic  of  .8681.  The  p-value  associated 
with  this  test  statistic  was  .0005  indicating  that  this  distribution  was  non-normal  as 
well. 

Both  of  the  populations  to  be  compared  appear  to  have  non-normal 
distributions  at  the  90  percent  level  of  confidence.  This  is  evident  graphically  as  well. 
The  problem  of  skewness  and  kurtosis  in  the  two  distributions,  as  shown  in  Figures  7 
and  8,  provides  graphical  support  of  the  non-normality  results  obtained  from  the 
Shapiro-Wilk  test.  As  such,  the  test  for  equal  variance  was  not  required,  and  the 
non-parametric  Mann-Whitney  test  was  selected  to  test  the  hypothesis. 

Results.  The  Mann-Whitney  model  tests  the  null  hypothesis  which  states  that 
the  two  sampled  populations  have  identical  probability  distributions  against  the 
alternate  hypothesis  which  states  that  the  probability  distribution  for  population  A  (the 
population  of  development  contracts  completed  before  Packard)  is  shifted  to  the  right 
of  that  for  population  B  (the  population  of  development  contracts  completed  after 
Packard).  The  test  performed  on  the  two  distributions  produced  a  test  statistic  of 
559.5  which  has  an  associated  p-value  of  .0153433.  As  the  level  of  significance  for 
this  test  was  established  at  10  percent  (a=.10)  the  results  suggest  that  sufficient 
evidence  exists  to  reject  the  null  hypothesis  at  the  90  percent  level  of  confidence. 

The  two  distributions  may  differ  in  location.  In  fact,  the  distribution  of  the  population 
of  development  contracts  completed  after  Packard  seems  to  be  shifted  to  the  left  of 
the  earlier  population.  Otherwise  stated,  the  mean  percentage  overruns  seems  to 
have  worsened  after  the  policy  went  into  effect. 
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Figure  7.  Histogram  -  Population  of  Development  Contracts  Before  Packard 
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Figure  8.  Histogram  -  Population  of  Development  Contracts  After  Packard 
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Data  Description  of  The  Phase  2  -Test  2  Model.  The  null  hypothesis  for 
the  second  test  states  that  the  mean  FO%  of  production  contracts  completed  before 
the  treatment  date  is  equal  to  the  mean  FO%  of  production  contracts  completed 
afterwards  while  the  alternate  hypothesis  states  that  the  FO%  before  is  different  than 
that  afterwards.  Table  9  displays  the  two  populations  of  contracts  examined  in  the 
first  test.  This  test  compared  the  population  of  production  contracts  completed 
before  the  Packard  recommendations  with  the  population  of  production  contracts 
completed  aftenwards.  Figures  9  and  10  show  a  graphical  representation  of  the  two 
populations  in  the  form  of  two  histograms.  A  smooth  line  representing  a  normal 
probability  distribution  is  overlaid  on  both  distributions  allowing  the  reader  to 
graphically  compare  the  actual  distributions  with  approximately  normal  distributions. 

Test  of  Normality  Results.  The  null  hypothesis  for  the  Shapiro-Wilk  test 
states  that  the  distribution  is  normal  while  the  alternate  hypothesis  states  that  it  is 
non-normal.  The  only  assumption  required  for  the  test  is  that  the  sample  is  a 
random  sample,  and  the  level  of  significance  (a)  for  the  normality  test  was  set  at  10 
percent.  The  Shapiro-Wilk  test  performed  on  the  population  of  production  contracts 
completed  before  Packard  produced  a  test  statistic  of  .8659.  The  p-value  associated 
with  this  test  statistic  was  approximately  zero  indicating  non-normality  in  the 
distribution.  The  Shapiro-Wilk  test  performed  on  the  population  of  contracts 
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completed  after  Packard  produced  a  test  statistic  of  .8268.  The  p-value  associated 
with  also  approximately  zero  indicating  that  this  distribution  was  non-normal  as  well. 

Both  of  the  populations  to  be  compared  appear  to  have  non-normal 
distributions  at  the  90  percent  level  of  confidence.  This  is  evident  graphically  as  well. 
The  problem  of  skewness  and  kurtosis  in  the  two  distributions,  as  shown  in  Figures  9 
and  10,  provides  graphical  support  of  the  non-normality  results  obtained  from  the 
Shapiro-Wilk  test.  As  such,  the  test  for  equal  variance  was  not  required,  and  the 
non-parametric  Mann-Whitney  test  was  selected  to  test  the  hypothesis. 

Results.  The  Mann-Whitney  model  tests  the  null  hypothesis  which  states  that 
he  two  sampled  populations  have  identical  probability  distributions  against  the 
alternate  hypothesis  which  states  that  the  probability  distribution  for  population  A  (the 
population  of  production  contracts  completed  before  Packard)  is  shifted  to  the  right  of 
that  for  population  B  (the  population  of  production  contracts  completed  after 
Packard).  The  test  performed  on  the  two  distributions  produced  a  test  statistic  of 
4191 .5  which  has  an  associated  p-value  of  .294025.  As  the  level  of  significance  for 
this  test  was  established  at  10  percent  (a=.10)  the  results  suggest  that  insufficient 
evidence  exists  to  reject  the  null  hypothesis  at  the  90  percent  level  of  confidence. 

The  two  distributions  may  not  differ  in  location.  In  other  words,  the  mean  percentage 
overrun  for  contracts  completed  after  the  treatment  doesn’t  appear  to  be  statistically 
different  than  the  mean  percentage  overrun  for  the  earlier  group.  No  effect  was 
noted. 
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Table  9.  Statistical  Analysis  Summary 


Tota 


Tota 


Tota 


1  population  of  contracts  : 

sample  1: 

Total  Pre 

148  values 

ranging  from 

-57.8947 

to 

12.3779 

sample  2: 

Total  Post 

121  values 

ranging  from 

-71.0526 

to 

25 

sample  1 

sample  2 

Shapiro-Wilk 

.846402 

Shapiro-Wilk 

.83301 

p-  value 

0 

p-  value 

0 

mean  FO% 

-5.56 

mean  FO% 

-9.58 

standard  error 

12.65 

standard  error 

17.42 

Mann-Whitney 
test  statistic 

7941 

p-value: 

0.0553347 

1 

population  of  development  contracts 

sample  1 : 

Dev  Pre 

47  values 

ranging  from 

-48.4848 

to 

9.52381 

sample  2; 

Dev  Post 

34  values 

ranging  from 

-71.0526 

to 

10.5263 

sample  1 

sample  2 

Shapiro-Wilk 

.807295 

Shapiro-Wilk 

.868057 

p-  value 

0 

p-  value 

.0005 

mean  FO% 

-4.14 

mean  FO% 

-15.29 

standard  error 

9.11 

standard  error 

20.46 

Mann-Whitney 
test  statistic 

569 

p-value: 

0.0140119 

1 

population  of 

production  contracts 

sample  1: 

Prod  Pre 

101  values 

ranging  from 

-57.8947 

to 

12.3779 

sample  2: 

Prod  Post 

87  values 

ranging  from 

-68.75 

to 

25 

sample  1 

sample  2 

Shapiro-Wilk 

Shapiro-Wilk 

.826806 

p-  value 

0 

p-  value 

0 

mean  FO% 

-6.22 

mean  FO% 

-7.24 

standard  error 

13.99 

standard  error 

15.66 

Mann-Whitney 
test  statistic 

4191.5 

p-value: 

0.294025 

1 
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Summary 

The  results  of  the  three  hypothesis  tests  are  summarized  in  Table  9.  The 
non-parametric  Mann-Whitney  test  was  used  in  all  of  the  tests  as  the  results  of 
the  Shapiro-Wilk  test  indicated  that  all  six  of  the  samples  appeared  to  have  non¬ 
normal  distributions.  A  graphical  analysis  of  the  distributions  supported  the  non- 
graphical  test  results  as  skewness  and  kurtosis  was  evident  in  all  six  samples. 
Variance  tests  were  not  required  as  a  basis  for  the  Mann-Whitney  model. 

The  results  of  the  first  hypothesis  test  indicated  that  the  mean  final 
overrun  percentage  (FO%)  for  contracts  completed  after  the  Packard 
Commission’s  recommendations  was  significantly  different  than  that  for  those 
completed  prior  to  the  treatment  at  the  .10  level  of  significance.  The  p-value  for 
the  Mann-Whitney  test  statistic  was  .055  which  is  a  significant  effect.  There  was 
sufficient  evidence  to  indicate  that  the  final  overrun  percentage  for  contracts 
completed  after  the  treatment  was  statistically  different  than  the  FO%  of  the 
earlier  population.  The  test  results  suggest  that  cost  performance  actually 
worsened  after  the  treatment  went  into  effect.  In  fact,  the  mean  percentage 
overrun  for  the  latter  group  (-9.58)  was  nearly  double  that  of  the  earlier  group  (- 
.56),  a  difference  of  4.02.  A  similar,  but  more  dramatic,  result  was  found  in  the 
second  test.  A  significant  difference  was  found  in  the  second  hypothesis  test 
regarding  development  contracts.  The  results  of  that  test  were  stronger  as  the  p- 
value  of  the  Mann-Whitney  test  statistic  was  .01 ,  considerably  less  than  the  .  1 0 
level  set  for  the  experiment.  Those  results  suggested  that  the  mean  FO%  for 
development  contracts  had  also  changed  in  the  post-treatment  group.  In  fact, 
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cost  performance  was  considerably  poorer  as  the  mean  FO%  more  than  tripled 
(-4.14  to  -15.29)  after  the  changes  went  into  effect,  a  difference  of  1 1.15. 

Lastly,  the  results  of  the  third  test  suggested  that  the  there  was  no  significant 
difference  in  the  FO%  for  production  contracts  completed  before  and  after  the 
treatment. 

Figures  11,12,  and  13  graphically  illustrate  these  results  using  overlaid 
density  traces  of  each  of  the  populations.  A  density  trace  is  a  plot  of  the  shape 
or  distribution  of  a  data  set,  especially  the  variations  in  density  over  the  range  of 
the  data.  The  dashed  lines  represent  the  population  distributions  of  the  Final 
Overrun  percentages  prior  to  the  change  while  the  solid  lines  represent  the  latter 
FO%  distributions.  The  top  graph,  Figure  11,  compares  the  distributions  of  the 
total  group  of  contracts  before  and  after  the  change.  The  post  group  appears  to 
have  shifted  to  the  left  slightly  providing  further  support  to  the  conclusions  drawn 
above.  Recall  that  a  leftward  shift  indicates  poorer  cost  performance  as  a 
negative  overrun  figure  signifies  a  cost  overrun.  The  middle  graph.  Figure  12, 
illustrates  the  second  test:  the  comparison  of  development  contracts  before  and 
after  the  change.  A  more  pronounced  leftward  shift  can  be  noted  in  the  post 
group  which,  again,  supports  the  conclusions  drawn  above.  Finally,  the  bottom 
graph.  Figure  13,  displays  the  comparison  of  the  distribution  of  production 
contracts  before  and  after  the  change.  As  concluded  above,  it  appears  no 
significant  change  has  occurred.  Table  10  on  page  79  summarizes  the  final 
results  of  the  three  hypothesis  tests. 
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Variables 

- TOTPRE 

-  TOTPOS 


Figure  11.  Overlaid  Density  Trace:  Total  Contracts 


Density  Traces 


Variables 
--  -  DEVPRE 
-  DEVPOS 


Figure  12.  Overlaid  Density  Trace:  Development  Contracts 
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Figure  13.  Overlaid  Density  Trace:  Production  Contracts 
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Table  10.  Summary  of  Hypothesis  Tests 


I  Hypothesis 

Results  &  Support 

H1: 

m03n  FOX)  (FOX)|l)  before  Dec  31  >  1991  “  FO  X)[X  afterwards 

reject 

p=.055  for  Mann- 
Whitney  test 

H2; 

FO  X)p.^gyglQpp.^gnt/before  ""  FO  X)p,jjgye|Qpp^gpyafter 

reject 

p=.014  for  Mann- 
Whitney  test 

H3; 

FO  X)^pj.gjjug^jQf^/j^fQf.g  --  PO  X)^p,.Q^jug{jQPJ/afJgP 

do  not  reject 
p=.294  for  Mann- 
Whitney  test 

This  concludes  the  presentation  of  the  results  of  the  hypothesis  tests 
proposed  in  chapter  three.  The  next  and  final  chapter  draws  conclusions  from 
these  statistical  tests  and  further  discusses  the  significance  of  the  findings  to  the 
Department  of  Defense.  Additionally,  recommendations  for  further  research  are 
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V.  Discussion 


Overview 

This  chapter  reviews  the  results  of  the  analysis  presented  in  Chapter  IV 
and  discusses  the  findings  of  this  research  effort  as  they  apply  to  the 
Department  of  Defense.  Additionally,  potential  limitations  of  this  study  are 
discussed  and  recommendations  are  made  for  further  research. 

Review  of  the  Hypotheses 

The  hypotheses  proposed  in  Chapter  I  are  restated  below: 

1 .  Given  a  sample  of  completed  contracts  (the  DAES  database),  the  final  overrun 
percentage  for  contracts  completed  after  Dec.  31,  1991  is  the  same  as  the 
percent  overrun  of  contracts  completed  before  that  date. 

2.  Given  a  sample  of  completed  development  contracts  (the  DAES  database), 
the  final  overrun  percentage  for  contracts  completed  after  Dec.  31,  1991  is  the 
same  as  the  percent  overrun  of  contracts  completed  before  that  date. 

3.  Given  a  sample  of  completed  production  contracts  (the  DAES  database),  the 
final  overrun  percentage  for  contracts  completed  after  Dec.  31,  1991  is  the  same 
as  the  percent  overrun  of  contracts  completed  before  that  date. 

These  hypotheses  were  tested  using  a  sample  of  269  contracts  extracted 
from  the  DAES  database  and  which  were  completed  between  January  1,  1988 
and  December  31 ,  1995.  This  eight  year  period  provided  for  approximately  four 
years  of  pre  and  post  treatment  data,  and  the  length  of  time  should  have 


mitigated  any  potential  bias  due  to  factors  such  as  fluctuations  in  the  defense 
business  cycle.  The  testing  was  conducted  in  order  to  test  the  effectiveness  of 
the  Packard  Commission’s  recommendations  on  reducing  cost  overruns  in  DoD 
acquisition  programs. 

The  principal  statistical  analysis  used  in  this  study  tested  the  three 
hypothesis  for  differences  in  their  population  means  as  this  would  allow  for 
inferences  to  be  made  about  the  differences  between  their  population 
parameters.  However,  prior  to  conducting  the  principal  test  all  of  the  sub¬ 
samples  were  individually  tested  for  normality  and  for  homogeneity  of  variance  in 
their  distributions.  The  results  of  these  tests  determined  whether  the  parametric 
t-test  or  the  nonparametric  Man-Whitney  test  would  be  used  as  the  determinant 
model.  All  of  the  sub-samples  in  this  study  appeared  to  have  non-normal 
distributions;  therefore,  the  three  hypotheses  were  tested  using  the 
nonparametric  Mann-Whitney  test  at  a  90  percent  level  of  confidence. 

Conclusions 

The  first  two  hypotheses  were  rejected  at  the  90  percent  level  of 
confidence.  The  results  of  the  first  two  tests  indicated  that  the  mean  final 
overrun  percentage  (FO%)  for  all  contracts  and  for  development  contracts 
completed  after  Dec.  31 ,  1991  was  different  than  that  for  those  completed  prior 
to  the  treatment.  Mean  percentage  overruns  were  actually  lower  after  the  policy 
went  into  effect  indicating  that  cost  performance  had,  in  fact,  worsened.  Recall 
that  a  negative  percentage  overrun  indicates  a  cost  overrun.  The  third 
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hypothesis  was  held  to  be  true  at  the  95  percent  level  of  confidence.  The  mean 
percentage  overrun  for  production  contracts  didn’t  appear  to  change  after  the 
treatment.  The  data  also  suggests  that  the  more  significant  change  evident 
between  the  distributions  of  development  contracts  may  account  for  the  effect 
noted  between  the  distributions  of  total  contracts. 

Overall,  the  testing  suggests  that  the  Commission’s  recommendations 
may  have  had  a  negative  effect  towards  reducing  cost  overruns  in  major  DoD 
acquisition  programs.  It  appears  that  cost  performance  actually  worsened  after 
the  treatment:  in  fact,  the  mean  percentage  overrun  for  total  contracts  nearly 
doubled  after  the  change.  The  most  dramatic  effect,  however,  was  noted  in  the 
population  of  development  contracts.  The  mean  percentage  overrun  for 
development  contracts  more  than  tripled  following  the  change  in  policy. 

Discussion  of  Findings 

The  results  of  these  tests  lie  counter  to  the  propositions  and  the  two 
concerns  introduced  in  the  Chapter  II  summary.  Based  on  the  information 
presented  in  the  literature  review  it  was  proposed  that  the  recommendations 
would  have  little  to  no  effect  on  reducing  cost  overruns.  Specifically,  it  was 
proposed  that  cost  overruns  for  contracts  completed  before  and  after  the  change 
would  essentially  be  the  same.  Furthermore,  it  was  proposed  that  the 
Commission’s  recommendations  would  affect  contracts  differently  based  on 
whether  the  contract  was  for  development  or  production.  In  addition  to  these 
proposals  two  major  concerns  were  forwarded  in  that  summary.  First  of  all,  it 
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was  noted  in  chapter  two  that  little  progress  has  been  made  over  the  years  in 
reducing  the  problem  of  cost  growth.  In  fact,  studies  have  indicated  that  cost 
growth  has  fluctuated  around  20  to  40  percent  since  the  mid  1960’s  and  little 
improvement  has  occurred  over  time.  As  the  Packard  recommendations  are 
similar  to  prior  reform  efforts  which  were  ineffective  why  should  the  DoD 
realistically  expect  improvements  in  cost  performance?  Secondly,  studies  have 
shown  that  the  minor  problems  of  poor  management  actually  account  for  only  a 
relatively  small  portion  of  the  cost  growth.  As  such,  even  if  the  recommendations 
were  effective  a  change  may  not  be  indicated  because  of  the  insignificance  of 
the  problem.  However,  the  results  of  this  thesis  suggest  that  the 
recommendations  may  have  had  a  significant  effect  on  cost  overruns,  albeit,  in 
the  wrong  direction.  Not  only  did  the  Commission’s  recommendations  appear  to 
have  inaccurately  targeted  some  of  the  possible  factors  leading  to  the  cost 
overrun  problem  but  it’s  possible  they  may  have  been  a  factor  leading  to  the 
poorer  cost  performance.  Another  possible  explanation  may  be  that  while  the 
policy  didn’t  directly  cause  poorer  cost  performance  it  may  have  created  an 
environment,  simply  by  it’s  ineffectiveness,  in  which  other  causal  factors  may 
have  been  allowed  to  operate. 

The  cost  performance  problem  seems  to  be  the  most  profound  in 
development  contracts.  Assuming  that  other  extraneous  causal  factors  have 
been  controlled  for  it’s  possible  that  the  Packard  Commission’s 
recommendations  may  have  actually  led  to  or  caused  poorer  cost  performance. 
Why,  then,  was  an  effect  noted  in  development  contracts  but  not  in  production 
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contracts?  The  percentage  cost  overruns  for  development  contracts  more  than 
tripled  while  no  significant  difference  occurred  with  production  contracts.  As 
discussed  in  Chapter  II  the  major  provisions  of  the  policy  seem  to  apply  to  both 
development  and  production  contracts  with  the  exception  of  the  requirement  for 
increased  testing  and  prototyping.  It  is  possible  that  this  recommendation  may 
account  for  the  dramatic  change  in  development  contract  cost  performance.  It’s 
reasonable  to  believe  that  the  recommendation  would  impact  development 
contracts  significantly  more  than  production  contracts. 

Unfortunately,  some  questions  remain  unanswered.  Had  the  policy 
focused  on  factors  such  as  planning  difficulties,  poor  estimating,  or  the  problems 
associated  with  competitive  bidding  perhaps  significant  reductions  may  have 
been  achieved.  At  minimum,  perhaps  this  type  of  approach  may  have  controlled 
the  overrun  problem  and  prevented  the  escalation.  The  factors  mentioned 
above  are  the  those  factors  categorized  as  estimation  errors  in  Chapter  II.  A 
number  of  possible  factors  have  been  identified  as  affecting  cost  growth. 
Independently  these  factors  may  lead  to  poor  cost  performance,  but  usually  one 
or  more  (or  a  combination  of  many)  are  the  real  causes  (Scott,  1983:38)  Future 
policies  may  have  more  success  if  estimation  errors  are  targeted  or  if,  perhaps,  a 
combination  of  estimation  errors  and  mismanagement  factors  are  targeted. 

The  results  also  support  the  proposition  discussed  in  chapter  three  which 
stated  that  the  policy  changes  would  probably  have  a  greater  impact  on 
contracts  in  the  development  phases  than  those  in  the  production  phases.  As 
systems  progress  from  the  development  to  production  phases  the  performance. 


schedule,  and  cost  parameters  become  more  predictable  and  less  variable  as 
the  risks  and  uncertainties  decrease  overtime.  Program  management  is  more 
effective  as  risks  and  uncertainties  are  reduced  over  time;  therefore,  one  would 
expect  the  larger  portion  of  cost  overruns  caused  my  management  inefficiencies 
to  occur  in  the  development  phases.  Consequently,  a  more  pronounced  effect 
was  noted  in  the  development  phase  whereas  no  effect  appeared  to  have 
occurred  in  the  production  phases. 

This  study  raises  some  concerns  regarding  the  appropriateness  of  current 
acquisition  reform  policies.  Presently,  the  DoD  is  admittedly  operating  in  an  era 
of  acquisition  reform.  The  passing  of  the  Federal  Acquisition  Streamlining  Act 
(FASA)  in  1994  and  the  Federal  Acquisition  Improvement  Act  (nicknamed  FASA 
II)  in  1995  marked  the  first  major  rewrites  of  acquisition  policy  since  the  Packard 
Commission’s  recommendations.  A  review  of  these  acts  indicates  striking 
similarities  between  their  major  provisions  and  those  of  earlier  reform  efforts 
including  the  Packard  Commission  study.  Themes  such  as  streamlining, 
decentralization  of  authority,  empowerment,  and  organizational  culture  change 
simply  re-emerge  in  a  newly  packaged  policy.  Once  again,  it  appears  that  the 
DoD  is  targeting  alleged  management  inefficiencies  as  it’s  principle  problem 
leading  to  procurement  inefficiency.  As  these  current  provisions  are  so  similar  to 
prior  reform  efforts  which  were  ineffective  the  DoD  should  not  realistically  expect 
improvements  in  cost  performance. 
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Limitations 


The  data  suggest  that  the  recommendations  fonwarded  by  the  Packard 
Commission  may  have  been  ineffective  in  reducing  cost  overruns  at  the  95 
percent  level  of  confidence;  infact,  the  policy  changes  may  have  actually  led  to 
or  fostered  poorer  cost  performance.  However,  a  number  of  limitations  must  be 
addressed  regarding  these  conclusions. 

Study  Design.  This  study  attempts  to  demonstrate  a  causal  relationship 
between  the  policy  changes  (the  recommendations  made  by  the  Commission) 
and  cost  overruns.  “The  essential  element  of  causation  is  that  A  “produces”  or  A 
“forces”  B  to  occur”  (Cooper  &  Emory,  1995:123).  However,  one  can  never 
empirically  demonstrate  with  certainty  that  some  variable  A  causes  some 
variable  B  to  occur.  At  best  one  can  structure  a  study  and  gather  supportive 
evidence  in  such  a  way  that  increases  the  probability  that  A  causes  B.  While  this 
study  can  demonstrate  that  the  time  order  of  events  moves  in  the  hypothesized 
direction  one  of  the  major  problems  with  it’s  design  is  that  it  was  merely 
impossible  to  eliminate  other  possible  causes. 

This  study  is  designed  as  an  ex  post  facto  design  particularly  because  it 
was  the  only  feasible  approach.  However,  ex  post  facto  designs  are  particularly 
weak  at  establishing  or  demonstrating  causality  (Cooper  &  Emory,  1995:129- 
131).  Since  archival  data  is  used  one  cannot  manipulate  the  variables  nor 
assign  subjects  to  treatment  and  control  groups.  As  a  result,  multiple  factors 
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may  be  the  cause  of  the  reduction  in  cost  overruns  rather  than  the  policy  change 
itself.  Furthermore,  this  study  used  a  one-group  pretest-posttest  design  which  is 
considered  a  weak  design  for  demonstrating  causality  as  it  fails  to  adequately 
control  threats  to  internal  validity.  The  results  must  be  interpreted  carefully  in 
order  to  avoid  post  hoc  fallacy. 

Threats  to  External  Validity.  The  process  by  which  the  contracts  were 
selected  may  threaten  external  validity  and  bias  the  results.  The  contracts 
examined  in  this  study  represent  only  a  portion  of  those  contracts  contained  in 
the  DAES  database.  First  of  all,  while  the  DAES  database  does  contain  cost 
performance  data  reported  on  sensitive  and  highly  classified  contracts  that  type 
of  data  is  not  included  in  this  study.  Secondly,  any  contracts  that  may  have  been 
canceled  prior  to  the  75  percent  completion  point  would  be  excluded  from  the 
sample.  Furthermore,  one  should  understand  that  the  definition  relied  upon  for  a 
“completed  contract”  reduces  the  usable  database.  Only  269  out  of  the  1,843 
individual  contracts  contained  in  the  database  were  retained  for  examination. 
Finally,  any  contracts  not  reporting  BAG  and  ACWP  figures  or  specifying 
program  phase  have  been  eliminated  from  the  population  as  well.  Thus,  some 
selection  bias  is  present  in  the  study.  As  such,  the  sample  used  in  this  research 
does  not  represent  the  entire  population  of  significant  DOD  contracts  and 
subcontracts  within  all  acquisition  programs  as  defined  in  DODI  5000.2.  One 
can  not  assume  then  that  any  effects  identified  in  this  sample  would  be 
representative  of  the  entire  population. 
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Threats  to  Internal  Validity.  Several  threats  to  internal  validity  weaken 


the  establishment  of  a  causal  relationship  between  the  policy  changes  and  the 
reduction  in  cost  overruns.  First  of  all,  a  history  effect  may  be  present.  Certain 
events  may  have  occurred  over  the  eight  year  time  period  in  which  the  contracts 
were  managed  that  may  confuse  the  relationship  being  studied.  The  contracts 
used  in  this  study  may  have  been  effected  by  political,  cultural-social,  or 
technological  changes.  Secondly,  one  can  reasonably  assume  that  a  maturation 
effect  would  be  present.  It’s  reasonable  to  believe  that  changes  may  have 
occurred  within  the  contract  that  are  a  function  of  the  passage  of  time,  learning, 
and  maturing  and  are  not  specific  to  any  particular  event.  Finally,  an 
instrumentation  effect  may  be  present.  “This  is  a  threat  to  internal  validity  that 
results  from  changes  between  observations,  in  measuring  instrument  or 
observer”  (Cooper  &  Emory,  1995:358).  This  study  examined  the  cost 
performance  of  269  contracts  as  reported  by  program  managers  in  the  DAES 
database  through  cost  performance  reports.  The  integrity  or  reliability  of  the 
reported  data  can  be  assumed  under  EVMS  reporting  requirements,  but  it 
cannot  be  controlled  for.  It’s  important  to  note  that  an  ex  post  facto/one-group 
pretest-posttest  design  such  as  the  one  used  in  this  study  is  considered  a  weak 
design  for  controlling  these  threats  to  internal  validity  (Cooper  &  Emory, 
1995:361). 


88 


Summary 


This  study  showed  that  the  recommendations  made  by  the  Packard 
Commission  may  have  been  ineffective  in  reducing  cost  overruns  in  major  DoD 
acquisition  programs  and  may  have  actually  led  to  or  fostered  poorer  cost 
performance.  The  data  also  suggests  that  the  recommendations  may  have  led 
to  or  fostered  higher  cost  overruns  in  the  riskier  development  phases  as  opposed 
to  production  phases. 

Recommendations  for  Further  Research 

It  is  recommended  that  future  research  investigate  other  possible  causes 
of  the  dramatic  change  in  cost  performance  after  December  31 ,  1991 .  Perhaps 
other  significant  factors  not  accounted  for  in  this  thesis  may  have  been 
responsible  for  the  changes  noted.  It  is  also  recommended  that  future  research 
include  investigation  of  more  recent  policy  changes  and  their  resultant  impact  on 
contract  cost  performance.  The  Federal  Acquisition  Streamlining  Act  (FASA)  of 
1994  and  the  Federal  Acquisition  Improvement  Act  (nicknamed  FASA  II)  of  1995 
are  considered  the  first  major  rewrites  of  government  procurement  regulations  in 
a  decade.  Investigation  into  the  tenets  of  these  policies  and  their  similarities  and 
differences  to  prior  reform  efforts  may  yield  insight  as  to  the  appropriateness  of 
the  current  path  of  acquisition  reform  policies.  This  investigation  may  also 
provide  an  evaluation  of  the  potential  of  these  current  policies  to  reduce  or 
control  cost  overruns  in  DoD  acquisition  programs. 
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Furthermore,  it  is  recommended  that  future  research  investigate  the 
effects  that  the  Integrated  Baseline  Review  (IBR)  policy  may  be  having  on 
reducing  cost  overruns.  This  1994  policy  is  a  major  variance  analysis  policy 
which  focuses  more  on  areas  such  as  planning  difficulties  and  estimation  errors 
as  opposed  to  mismanagement.  There  is  some  early  indication  that  since  IBR’s 
inception  in  1994  the  overall  performance  in  terms  of  cost  and  schedule 
performance  has  improved  for  a  number  of  programs  (Christensen,  1997).  It 
may  take  two  or  three  more  years  before  sufficient  data  becomes  available  to 
test  for  possible  effects,  but  the  information  presented  in  this  thesis  suggests  that 
the  IBR  policy  may  have  more  success  at  reducing  cost  overruns  than  the 
Packard  Commission’s  recommendations  as  the  policy  seems  to  target  those 
factors  accounting  for  the  relatively  larger  portion  of  cost  gro\Arth.  Finally,  it  is 
recommended  that  future  research  replicate  this  study  using  the  cost  growth 
model.  A  comparison  of  the  two  models  may  provide  policy  makers  with  a 
broader  interpretation  of  policy  effectiveness. 
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Appendix:  Data  Table  For  Contracts  Used  In  This  Study 


SUBMITDATE 

SERVICE 

PHASE 

ACWP 

BAG 

4/25/88 

Navy 

P 

34 

33.00 

-1 

-3.030 

89.189 

5/25/88 

Navy 

P 

657 

725.00 

68 

9.379 

90.852 

8/25/88 

Army 

P 

117 

76.00 

-41 

-53.947 

97.436 

8/25/88 

Army 

P 

58 

49.00 

-9 

-18.367 

90.741 

10/25/88 

Navy 

P 

204.00 

-13 

-6.373 

88.696 

12/25/88 

Air  Force 

D 

22 

20.00 

-2 

-10.000 

90.909 

3/25/89 

Navy 

D 

101 

99.00 

-2 

-2.020 

76.744 

5/25/89 

Air  Force 

D 

114 

123.00 

9 

7.317 

97.619 

5/25/89 

Navy 

P 

30 

19.00 

-11 

-57.895 

90.476 

6/25/89 

Air  Force 

D 

41 

43.00 

2 

4.651 

95.556 

6/25/89 

Air  Force 

D 

31 

32.00 

1 

3.125 

94.118 

6/25/89 

Air  Force 

D 

218 

220.00 

2 

0.909 

93.220 

6/25/89 

Air  Force 

D 

19 

21.00 

2 

9.524 

80.769 

6/25/89 

Air  Force 

D 

43 

45.00 

2 

4.444 

78.947 

6/25/89 

Air  Force 

P 

43 

38.00 

-5 

-13.158 

102.703 

6/25/89 

Air  Force 

P 

75 

64.00 

-11 

-17.188 

100.000 

6/25/89 

Air  Force 

P 

97 

96.00 

-1 

-1.042 

98.969 

6/25/89 

Air  Force 

P 

72 

69.00 

-3 

-4.348 

97.183 

6/25/89 

Air  Force 

P 

95 

91.00 

-4 

-4.396 

96.809 

6/25/89 

Air  Force 

P 

82 

90.00 

8 

8.889 

95.745 

6/25/89 

Air  Force 

P 

40 

44.00 

4 

9.091 

95.652 

6/25/89 

Air  Force 

P 

76 

75.00 

-1 

-1.333 

94.937 

6/25/89 

Air  Force 

P 

163 

166.00 

3! 

1.807 

94.318 

6/25/89 

Air  Force 

P 

62 

66.00 

4 

6.061 

92.958 

6/25/89 

Air  Force 

P 

320 

275.00 

^5 

-16.364 

92.905 

6/25/89 

Air  Force 

P 

62 

60.00 

-2 

-3.333 

92.308 

6/25/89 

Air  Force 

P 

116 

120.00 

4 

3.333 

91.603 

6/25/89 

Air  Force 

P 

148 

130.00 

-18 

-13.846 

90.909 

6/25/89 

Army 

P 

750 

792.00 

42 

5.303 

94.511 

6/25/89 

Army 

P 

667 

709.00 

42 

5.924 

92.923 

6/25/89 

Navy 

D 

42 

42.00 

0 

0.000 

100.000 

6/25/89 

Navy 

D 

87 

82.00 

-5 

-6.098 

93.182 

6/25/89 

Navy 

D 

843 

844.00 

1 

0.118 

92.544 

7/25/89 

Air  Force 

P 

70 

7 

9.091 

106.944 

7/25/89 

Air  Force 

P 

358 

331.00 

-27 

-8.157 

99.399 

7/25/89 

Air  Force 

P 

135 

130.00 

-5 

-3.846 

97.744 

7/25/89 

Air  Force 

P 

1128 

1073.00 

-55 

-5.126 

95.293 

7/25/89 

Air  Force 

P 

•  58 

54.00 

-4 

-7.407 

94.737 

7/25/89 

Air  Force 

P 

51 

50.00 

-1 

-2.000 

89.286 

7/25/89 

Army 

P 

184 

193.00 

9 

4.663 

95.545 

7/25/89 

Navy 

D 

161 

133.00 

-28 

-21.053 

97.794 

7/25/89 

Navy 

D 

139 

137.00 

-2 

-1.460 

93.836 

7/25/89 

Navy 

D 

139 

137.00 

-2 

-1.460 

93.836 
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SUBMtTDATE 

SERVICE 

PHASE 

ACWP 

BAC 

cv 

CV%  PC% 

7/25/89 

Navy 

D 

954 

907,00 

-47 

-5.182 

92.551 

7/25/89 

Navy 

D 

954 

907.00 

-47 

-5.182 

92.551 

7/25/89 

Navy 

D 

-4 

7/25/89 

Navy 

P 

148.00 

4 

2.703 

97.368 

7/25/89 

Navy 

P 

607.00 

-91 

-14.992 

95.141 

7/25/89 

Navy 

P 

114 

115.00 

1 

0.870 

93.496 

7/25/89 

Navy 

P 

130 

126.00 

-4 

-3.175 

93.333 

8/25/89 

Air  Force 

P 

12687 

12228.00 

-459 

-3.754 

96.072 

8/25/89 

Air  Force 

P 

592 

595.00 

3 

0.504 

93.260 

8/25/89 

Army 

D 

12 

13.00 

1 

7.692 

86.667 

8/25/89 

Navy 

D 

■BSI 

-2 

-5.000 

97.561 

8/25/89 

Navy 

P 

9 

2.486 

101.685 

8/25/89 

Navy 

P 

9 

2.486 

101.685 

8/25/89 

Navy 

P 

247.00 

0 

0.000 

100.407 

8/25/89 

Navy 

P 

216.00 

-23 

-10.648 

94.737 

8/25/89 

Navy 

P 

48.00 

-6 

-12.500 

94.118 

8/25/89 

Navy 

P 

269 

307.00 

38 

12.378 

90.828 

9/25/89 

Air  Force 

P 

150 

153.00 

3 

1.961 

91.617 

9/25/89 

Air  Force 

P 

86 

92.00 

6 

6.522 

91.089 

9/25/89 

Army 

D 

96 

98.00 

2 

2.041 

98,990 

9/25/89 

Navy 

D 

367 

366.00 

-1 

-0.273 

94.574 

9/25/89 

Navy 

P 

72 

76.00 

4 

5.263 

105.556 

9/25/89 

Navy 

P 

-12 

-50.000 

100.000 

9/25/89 

Navy 

P 

34 

0 

0.000 

100.000 

9/25/89 

Navy 

P 

870 

790.00 

-80 

-10.127 

97.411 

10/25/89 

Air  Force 

P 

69 

66.00 

-3 

-4.545 

95.652 

10/25/89 

Navy 

P 

88 

81.00 

-7 

-8.642 

96.429 

10/25/89 

Navy 

P 

48 

40.00 

-8 

-20.000 

93.023 

11/25/89 

Air  Force 

D 

596 

608.00 

12 

1.974 

96.970 

11/25/89 

Air  Force 

P 

410 

415.00 

5 

1.205 

97.190 

11/25/89 

Army 

D 

121.00 

-21 

-17.355 

98.374 

11/25/89 

Army 

P 

23.00 

2 

8.696 

76.667 

12/25/89 

Army 

P 

60.00 

89.552 

12/25/89 

Navy 

D 

316 

290.00 

-26 

-8.966 

97.315 

12/25/89 

Navy 

P 

113 

105.00 

-8 

-7.619 

92.920 

2/25/90 

Army 

D 

62 

62.00 

0 

2/25/90 

Navy 

P 

52 

50.00 

-2 

3/25/90 

Air  Force 

P 

73 

81.00 

8 

3/25/90 

Air  Force 

P 

212 

213.00 

1 

0.469 

93.421 

3/25/90 

Air  Force 

P 

1084 

1084.00 

0 

0.000 

80.296 

3/25/90 

Navy 

P 

54 

54.00 

0 

0.000 

98.182 

4/25/90 

Air  Force 

D 

870 

780.00 

-90 

-11.538 

94.775 

4/25/90 

Air  Force 

P 

409 

324.00 

-85 

-26.235 

105.537 

4/25/90 

Air  Force 

P 

4/25/90 

Air  Force 

P 

152 

148.00 

-4 

-2.703 

90.798 
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ACWP 

BAG 

■33 

4/25/90 

Navy 

D 

227.00 

■BE 

■QBE 

4125190 

Navy 

P 

290 

265.00 

BEISS 

92.982 

6/25/90 

Air  Force 

P 

96 

109.00 

B 

BQQ^3 

100.000 

6/25/90 

Air  Force 

P 

84 

91.00 

7 

7.692 

100.000 

6/25/90 

Air  Force 

P 

156 

131.00 

-25 

-19.084 

97.761 

6/25/90 

Air  Force 

P 

-25 

-20.661 

96.032 

6/25/90 

Navy 

D 

HBBI 

1.258 

108.904 

6/25/90 

Navy 

P 

171 

168.00 

-3 

-1.786 

104.348 

7/16/90 

Air  Force 

P 

33 

36.00 

3 

8.333 

97.297 

8/25/90 

Navy 

D 

194 

188.00 

-6 

-3.191 

105.028 

9/25/90 

Air  Force 

D 

67 

66.00 

-1 

-1.515 

98.507 

9/25/90 

Air  Force 

P 

188.00 

-30 

-15.957 

92.157 

9/25/90 

P 

1631 

1716.00 

85 

4.953 

95.386 

10/25/90 

D 

41 

36.00 

-5 

-13.889 

92.308 

10/25/90 

P 

36 

40.00 

4 

10.000 

90.909 

10/25/90 

Navy 

D 

68 

69.00 

1 

1.449 

95.833 

10/25/90 

Navy 

P 

980 

881.00 

-99 

-11.237 

106.017 

P 

63 

63.00 

0 

0.000 

105.000 

10/25/90 

Navy 

P 

45 

46.00 

1 

2.174 

92.000 

10/25/90 

Navy 

P 

46 

38.00 

-8 

-21.053 

88.372 

11/25/90 

Army 

D 

42 

39.00 

-3 

-7.692 

100.000 

11/25/90 

Army 

D 

49 

33.00 

-16 

-48.485 

91.667 

11/25/90 

Navy 

P 

94.00 

-36 

-38.298 

100.000 

12/25/90 

Air  Force 

D 

160 

156.00 

-4 

-2.564 

200.000 

12/25/90 

Navy 

D 

3910 

3513.00 

-397 

-11.301 

101.123 

12/25/90 

Navy 

D 

684 

661.00 

-23 

-3.480 

100.609 

12/25/90 

Navy 

P 

1389 

1293.00 

-96 

-7.425 

96.927 

12/25/90 

Navy 

P 

336 

331.00 

-5 

-1.511 

89.459 

2/25/91 

Army 

p 

182 

165.00 

-17 

-10.303 

100.000 

2/25/91 

Navy 

D 

703 

662.00 

-41 

-6.193 

99.399 

3/25/91 

Air  Force 

P 

183 

154.00 

-29 

-18.831 

98.089 

3/25/91 

Air  Force 

P 

415 

432.00 

17 

3.935 

92.111 

3/25/91 

Navy 

P 

169 

162.00 

-7 

-4.321 

96.429 

3/25/91 

Navy 

P 

79 

81.00 

2 

2.469 

92.045 

4/25/91 

Air  Force 

P 

42 

36.00 

-6 

-16.667 

97.297 

6/25/91 

Air  Force 

D 

90 

86.00 

-4 

-4.651 

96.629 

6/25/91 

Air  Force 

D 

282 

280.00 

-2 

-0.714 

96.552 

6/25/91 

Air  Force 

P 

72 

82.00 

10 

12.195 

82.000 

6/25/91 

Navy 

D 

45 

46.00 

1 

2.174 

95.833 

6/25/91 

Navy 

D 

172 

172.00 

0 

0.000 

76.786 

6/25/91 

Navy 

P 

141 

140.00 

-1 

-0.714 

98.592 

6/25/91 

Navy 

P 

148 

142.00 

-6 

-4.225 

95.946 

7/25/91 

Air  Force 

P 

721 

707.00 

-14 

-1.980 

96.717 

7/25/91 

Air  Force 

P 

317 

267.00 

-50 

-18.727 

94.346 

7/25/91 

Navy 

P 

125 

93.00 

-32 

-34.409 

93.939 

93 


SUBMITDATE 

SERVICE 

PHASE 

BAC  CV 

cv% 

PC% 

7/25/91 

Navy 

P 

196 

182.00 

-14 

-7.692 

93.814 

8/25/91 

Air  Force 

D 

98 

90,00 

-8 

-8.889 

100.000 

8/25/91 

Navy 

P 

71 

57.00 

-14 

-24.561 

116.327 

9/25/91 

Air  Force 

D 

416 

393.00 

-23 

-5.852 

159.109 

9/25/91 

Navy 

P 

92 

96.00 

4 

4.167 

100.000 

10/25/91 

Air  Force 

P 

832 

800.00 

-32 

-4.000 

96.618 

10/25/91 

Navy 

P 

532 

548.00 

16 

2.920 

98.032 

10/25/91 

Navy 

P 

260 

187.00 

-73 

-39.037 

97.906 

10/25/91 

Navy 

P 

39 

36.00 

-3 

-8.333 

85.714 

11/25/91 

Navy 

P 

158 

145.00 

-13 

-8.966 

96.026 

P 

liHES 

HE 

-1.923 

95.413 

D 

IHBQ 

-1.235 

99.022 

D 

0 

0.000 

98.256 

12/25/91 

Navy 

P 

261 

189.00 

-72 

-38.095 

98.953 

12/25/91 

Navy 

P 

201 

166.00 

-35 

-21.084 

94.857 

1/25/92 

Air  Force 

D 

134 

139.00 

5 

3.597 

99.286 

1/25/92 

Air  Force 

D 

76 

64.00 

-12 

-18.750 

96.970 

1/25/92 

Air  Force 

D 

70 

61.00 

-9 

-14.754 

92.424 

1/25/92 

Air  Force 

P 

632 

580,00 

-52 

-8.966 

98.472 

1/25/92 

Army 

P 

413 

406.00 

-7 

-1.724 

91.031 

1/25/92 

Navy 

P 

72 

78.00 

6 

7.692 

98,734 

2/25/92 

Navy 

P 

139 

110,00 

-29 

-26.364 

96.491 

3/25/92 

Air  Force 

D 

118 

105.00 

-13 

-12.381 

82.677 

3/25/92 

Air  Force 

P 

67 

62.00 

-5 

-8.065 

98.413 

3/25/92 

Air  Force 

P 

315 

242.00 

-73 

-30.165 

97.581 

3/25/92 

Air  Force 

P 

268 

253.00 

-15 

-5.929 

95.833 

3/25/92 

Navy 

D 

872 

876.00 

4 

0.457 

95.425 

4/15/92 

Navy 

P 

173 

119.00 

-54 

-45.378 

93.701 

4/25/92 

Army 

D 

308 

233.00 

-75 

-32.189 

132.386 

4/25/92 

Navy 

P 

1485 

1173.00 

-312 

-26.598 

99.239 

5/25/92 

Navy 

P 

173 

148.00 

-25 

-16.892 

98.667 

5/25/92 

Navy 

P 

573 

604.00 

31 

5.132 

97.419 

5/25/92 

Navy 

P 

573 

604.00 

31 

5.132 

97.419 

5/25/92 

Navy 

P 

217 

194.00 

-23 

-11.856 

96.517 

5/25/92 

Navy 

P 

348 

350.00 

2 

0.571 

93.085 

6/25/92 

Air  Force 

P 

408 

380.00 

-28 

-7.368 

98.701 

6/25/92 

Navy 

D 

204 

204.00 

0 

0.000 

91.071 

7/25/92 

Navy 

mm 

1256.00 

40 

3.185 

105.105 

7/25/92 

Navy 

819.00 

-4 

-0.488 

97.733 

9/25/92 

Air  Force 

P 

116 

118.00 

2 

1.695 

85.507 

10/25/92 

Navy 

D 

1936 

1781.00 

-155 

-8.703 

116.405 

10/25/92 

Navy 

D 

63 

39.00 

-24 

-61.538 

95.122 

^^||||||[||| 

P 

116 

115.00 

-1 

-0.870 

96.639 

P 

114 

124.00 

10 

8.065 

95.385 

12/25/92 

Air  Force 

D 

18 

18.00 

0 

0.000 

100.000 

94 


Essa 

isa 

CV% 

PC% 

■oa 

wm 

-3.876 

94.161 

12/25/92 

Navy 

P 

-2.273 

92.308 

1/25/93 

Air  Force 

D 

138 

-16.949 

129.670 

1/25/93 

Air  Force 

P 

165 

1/25/93 

Air  Force 

P 

150 

1/25/93 

Air  Force 

P 

223 

■■IQ 

1/25/93 

Air  Force 

P 

HBB 

■iVJAW 

1/25/93 

Air  Force 

P 

HEQI 

■■ 

1/25/93 

Navy 

P 

2/25/93 

Army 

D 

114 

92.00 

-22 

-23.913 

98.925 

2/25/93 

Navy 

P 

107 

103.00 

-4 

-3.883 

95.370 

2125193 

Navy 

P 

97 

-3 

3/25/93 

Air  Force 

D 

-23 

-25.275 

95.789 

3/25/93 

Navy 

P 

-43 

-8.222 

176.689 

3/25/93 

Navy 

P 

30 

34.00 

4 

11.765 

97.143 

3/25/93 

Navy 

P 

109 

115.00 

6 

5.217 

95.833 

3/25/93 

Navy 

P 

266 

222.00 

-44 

-19.820 

95.690 

4/25/93 

Air  Force 

P 

416 

404.00 

-12 

-2.970 

91.403 

5/25/93 

Army 

D 

65 

38.00 

-27 

-71.053 

100.000 

5/25/93 

Navy 

P 

216 

194.00 

-22 

-11.340 

97.000 

P 

720.00 

-144 

-20.000 

99.585 

6/25/93 

Air  Force 

P 

59.00 

-10 

-16.949 

96.721 

6/25/93 

Navy 

P 

8 

25.000 

100.000 

6/25/93 

Navy 

P 

74 

2 

2.632 

93.827 

7/25/93 

Navy 

P 

-19 

-2.408 

98.996 

8/25/93 

Navy 

P 

272 

222.00 

-50 

-22.523 

114.433 

8/25/93 

Navy 

P 

455 

513.00 

58 

11.306 

95.000 

8/25/93 

Navy 

P 

658 

626.00 

-32 

-5.112 

85.636 

9/25/93 

Navy 

P 

173 

174.00 

1 

0.575 

79.452 

10/25/93 

Air  Force 

P 

234 

236.00 

2 

0.847 

112.919 

10/25/93 

Navy 

P 

1475 

1400.00 

-75 

-5.357 

131.827 

10/25/93 

Navy 

P 

610 

9 

1.454 

101.309 

12/25/93 

Navy 

P 

482 

-24 

-5.240 

157.931 

1/25/94 

Navy 

D 

79 

81.00 

2 

2.469 

197.561 

1/25/94 

Navy 

P 

nog 

108.00 

-1 

-0.926 

100.000 

4/25/94 

Air  Force 

P 

iHsa 

102.00 

-4 

-3.922 

96.226 

4/25/94 

Navy 

1001.00 

-3 

-0.300 

98.234 

5/25/94 

Air  Force 

D 

1088 

1088.00 

0 

0.000 

124.485 

7/25/94 

Navy 

P 

1441 

1131.00 

-310 

-27.409 

97.668 

7/25/94 

Navy 

P 

901 

-105 

7/25/94 

Navy 

P 

635 

mm 

-27 

8/25/94 

Navy 

645.00 

-77 

-11.938 

117.060 

9/25/94 

Air  Force 

p 

115 

122.00 

7 

5.738 

91.045 

9/25/94 

Navy 

p 

110 

111.00 

1 

0.901 

96.522 

10/25/94 

Air  Force 

p 

1147 

958.00 

-189 

-19.729 

106.800 

95 


SUBMITDATE 

SERVICE 

PHASE 

ACWP 

BAC 

cv 

CV% 

PC% 

10/25/94 

Air  Force 

P 

984 

899.00 

-85 

-9.455 

95.132 

10/25/94 

Air  Force 

P 

1208 

741.00 

-467 

-63.023 

95.122 

10/25/94 

Air  Force 

P 

910 

577.00 

-333 

-57.712 

94.435 

11/25/94 

Air  Force 

D 

224 

225.00 

1 

0.444 

104.651 

11/25/94 

Air  Force 

P 

27 

16.00 

-11 

-68.750 

76.190 

11/25/94 

Navy 

P 

442 

468.00 

26 

5.556 

95.122 

12/25/94 

Air  Force 

P 

994 

866.00 

-128 

-14.781 

98.297 

12/25/94 

Air  Force 

P 

764 

728.00 

-36 

-4.945 

96.296 

12/25/94 

Air  Force 

P 

1478 

1372.00 

-106 

-7.726 

94.817 

12/25/94 

Air  Force 

P 

638 

550.00 

B2BSiE] 

12/25/94 

Air  Force 

P 

166 

150.00 

-16 

-10.667 

82.873 

12/25/94 

Army 

D 

129 

134.00 

5 

3.731 

84.810 

1/25/95 

Navy 

P 

1068 

1062.00 

-6 

-0.565 

96.898 

2/25/95 

Air  Force 

D 

76 

78.00 

2 

2.564 

90.698 

2/25/95 

Navy 

P 

816 

774.00 

-42 

-5.426 

90.632 

3/25/95 

Navy 

P 

500 

449.00 

-51 

-11.359 

155.903 

3/25/95 

Navy 

P 

93 

92.00 

-1 

-1.087 

77.966 

4/25/95 

Air  Force 

D 

8037 

5141.00 

-2896 

-56.331 

93.202 

4/25/95 

Air  Force 

D 

117 

83.00 

-34 

-40.964 

88.298 

4/25/95 

Air  Force 

D 

68 

76.00 

8 

10.526 

85.393 

4/25/95 

Air  Force 

D 

8987 

8467.00 

-520 

-6.141 

80.985 

4/25/95 

Air  Force 

P 

129 

87.00 

-42 

-48.276 

108.750 

4/25/95 

Air  Force 

P 

1371 

1340.00 

-31 

-2.313 

96.195 

4/25/95 

Air  Force 

P 

102 

98.00 

-4 

-4.082 

95.146 

4/25/95 

Air  Force 

P 

626 

576.00 

-50 

-8.681 

93.204 

4/25/95| 

Navy 

D 

17 

16.00 

-1 

-6.250 

94.118 

4/25/95 

Navy 

D 

20 

19.00 

-1 

-5.263 

79.167 

4/25/95 

Navy 

P 

1700 

1738.00 

38 

2.186 

91.763 

4/25/95 

Navy 

P 

548 

539.00 

-9 

-1.670 

81.667 

5/25/95 

Air  Force 

D 

631 

601.00 

-30 

-4.992 

84.648 

5/25/95 

Air  Force 

D 

100 

74.00 

-26 

-35.135 

75.510 

5/25/95 

Air  Force 

P 

106 

111.00 

5 

4.505 

96.522 

5/25/95 

Army 

D 

136 

141.00 

5 

3.546 

89.809 

5/25/95 

Army 

D 

136 

116.00 

-20 

-17.241 

76.316 

5/25/95 

Army 

P 

216 

222.00 

6 

2.703 

91.358 

5/25/95 

Navy 

D 

177 

164.00 

-13 

-7.927 

128.125 

5/25/95 

Navy 

P 

962 

1024.00 

62 

6.055 

97.431 

5/25/95 

Navy 

P 

446 

487.00 

41 

8.419 

92.586 

5/25/95 

Navy 

P 

894 

848.00 

-46 

-5.425 

89.641 

1 - ^ - 

6/25/95 

Navy 

D 

1092 

1066.00 

-26 

-2.439 

111.857 

6/25/95" 

Navy 

D 

673 

478.00 

-195 

-40.795 

98.354 

6/25/95 

Navy 

D 

53 

51.00 

-2 

-3.922 

91.071 

6/25/95 

Navy 

D 

55 

41.00 

-14 

-34.146 

80.392 

6/25/95 

Navy 

P 

501 

511.00 

10 

1.957 

90.764 

6/25/95 

Navy 

P 

999 

842.00 

-157 

-18.646 

90.733 

96 


SUBMITDATE 

SERVICE 

PHASE 

ACWP 

BAG 

CV 

CV%  PC% 

6/25/95 

Navy 

P 

251 

248.00 

-3 

-1.210 

82.119 

legend 

Phase:  P  =  production,  D  =  development 
CV;  cost  variance  (final  overrun) 

CV%:  cost  variance  percentage  (final  overrun  percentage) 
PC%:  percent  complete 
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